Page MenuHomePhabricator
Diviner AG Projects RTC Platforms Routes: Messaging Server

Routes: Messaging Server
AG Projects RTC Platforms (Routing Guides)

Messaging Server

Sylk Server provides online and offline messaging services for standard SIP end-points and WebRTC clients that support SylkRTC API.

The functionality is similar to what WhatsApp and Telegram offer with the difference that both client and server software are fully open source and based on open standards. Reference Web, Mobile, Desktop implementations and a live service are available for testing and interoperability purposes.

Sylk Server maintains a journal for each user with the messages and their modifications encapsulated using the following standards:

SIP offline messages can by synced between all devices by applying the journal of operations for each user.

For real time synchronization of online devices, standard SIP parallel forking is used.

The storage is transparent for end-users and it is up to the end-user devices to encrypt the payloads before sending them out by using an OpenPGP implementation.

Sylk Server has support for managing the public and private keys for PGP encrypted messages.

Features

  • Message replication between multiple devices
  • Offline storage when devices are not online
  • Message delivery and display reports
  • Message deletion, partial or per contact
  • Tracks remaining unread messages
  • Incremental journal per account
  • Distributed storage support using Cassandra
  • SIP and WebRTC APIs
  • Mobile push notifications
  • PGP public/private keys management

API

The functions are documented in this JavaScript API:

Sample clients

Complete fully featured clients implementing SylkRTC API:

Content types

Sylk Server handle in special ways different content types.

application/sylk-api-token

Allows a SIP device to retrieve an authentication token that can be later used for authorization to retrieve the journal using HTTP requests.

Example:

SENDING: Packet 60, +0:01:52.503136
2021-11-15 14:33:07.194842: 192.168.1.11:50768 -(SIP over TLS)-> 85.17.186.23:5061
MESSAGE sip:ag@sip2sip.info SIP/2.0
Via: SIP/2.0/TLS 192.168.1.11:50768;rport;branch=z9hG4bKPj8MUCkGXTF7nXCTYxcMHYL5tzC8p2j-;alias
From: <sip:ag@sip2sip.info>;tag=ePBmQXDzJCFfRZR3FekS3mTjCsapf
To: <sip:ag@sip2sip.info>
Call-ID: X6saJssTX.wmSZYDf2GrZXOAoGMiEBh0
User-Agent: Blink 8.7.0 (MacOSX)
Content-Type: application/sylk-api-token
Content-Length:    14

I need a token

Response:

RECEIVED: Packet 62, +0:01:52.816791
2021-11-15 14:33:07.508497: 85.17.186.23:5060 -(SIP over TCP)-> 192.168.1.11:50735
MESSAGE sip:46139205@192.168.1.11:50685;transport=tcp SIP/2.0
Via: SIP/2.0/TCP 85.17.186.23:5060;branch=z9hG4bKf3d6.33c61951.0;i=57fa5934
Via: SIP/2.0/TCP 85.17.186.26:35305;received=85.17.186.26;rport=35305;branch=z9hG4bKPj5eb777ff-0026-44b0-b17b-9d468fe76931;alias
From: "SylkServer" <sip:sylkserver@85.17.186.26>;tag=7449abda-1da-4df3-b8d1-0c94cb031f18
To: <sip:ag@sip2sip.info>
Call-ID: 3507f82d-f027-4d3a-bfb6-3efb4655caa1
User-Agent: SylkServer-6.0.0
Content-Type: application/sylk-api-token
Content-Length:   153

{"token": "iKQI7svOtyGkfsfsfsh9qGstVUOnTgrFxpo1ik40", "url": "https://webrtc-gateway.sipthor.net:9943/webrtcgateway/messages/history/ag@sip2sip.info"}

By sending a GET to the URL, one can retrieve the journal since the last id.

if last_id:
    url = "%s/%s" % (url, account.sms.history_last_id)     
req = urllib.request.Request(url, method="GET")
req.add_header('Authorization', 'Apikey %s' % account.sms.history_token)

Typically the request is sent every time the device connects to the network and then saves the last id that can be used later to resume the journal processing.

Sample code:

https://github.com/AGProjects/blink-cocoa/blob/master/SMSWindowManager.py

text/pgp-public-key

Once a user sends out a message with such payload, it will be stored in the user profile and end-points that implement public key fetch function can retrieve the key before starting a conversation.

Sample code:

https://github.com/AGProjects/sylkrtc.js/blob/master/lib/account.js

text/pgp-private-key

This is used for end-points to replicate their private keys, so multiple devices can by synchronized.

To replicate messages on multiple devices you need the same private key on all of them.

To be compatible with existing clients the message must be addressed to the same user and the payload must consist of the user PGP public key in clear text followed by the PGP encrypted with a symmetric code private key.

Example:

SENDING: Packet 88, +0:35:02.954481
2021-11-15 15:06:17.646187: 192.168.1.11:51541 -(SIP over TLS)-> 85.17.186.23:5061
MESSAGE sip:ubuntu@test.com SIP/2.0
Via: SIP/2.0/TLS 192.168.1.11:51541;rport;branch=z9Xb--OH3b91;alias
Max-Forwards: 70
From: <sip:ubuntu@test.com>;tag=pPOSjBJx2nm9ZG8DZX-ObDtCfDaqJBDk
To: <sip:ubuntu@test.com>
Call-ID: bhNsVTpEoFNNqXtNcpm
CSeq: 52490 MESSAGE
User-Agent: Blink 8.7.0 (MacOSX)
Content-Type: text/pgp-private-key
Content-Length:  5378

-----BEGIN PGP PUBLIC KEY BLOCK-----

xsFNBGDQz0cBEACw7ZCd2CIw5udWY5VOV4ZrzvyIdt8idBoUqbUJ6Lm55KVMj7Kh
6Lu13BKfV3m04Q3L3b5KboqIsbJLivlpg2YX+vOAAzEDE9BTZN0o+orDWxP1Dz9U
LvtloSRoNK1NZj+KkbKfRXUZCMnyXDiN64cYSFLGp0esT/DhyfI09p63dqegl+Ut
uenVLzne9PkpdewS0OUmfKUGBY86SPUewwBkFFVxYyIWQu+yJLfMt7jxTHnD3dcT
mDaRljMZj6qhzQ2F2sc68KHy2FIeXR9kOxMq8T6aEw0JwAA4rLWWCT6MLEtjDPng
cduwml6F7g==
=Xp4B
-----END PGP PUBLIC KEY BLOCK-----
-----BEGIN PGP MESSAGE-----
O+rMXHZq7rhMlYyBYJFSgVZTn2t58s/3NkTVewwvgstl10BUDgn9zBggElo86duB
2fqwEGCYSny1CLfWtMe/8GBrLd/pCpAWVlzsaI/s/7QbLqW8d0nq1uft8ubRvyWb
wIl0lLwuSReeyNcFHw53jpumFQbXYtZRM8UhhKmJoeKwZh++AIYUdG5OlPUb32hF
8mhmr1m1rhxPJTWQvaCsYg==
=sZwQ
-----END PGP MESSAGE-----

The receiving end-point must decrypt the message and save the PGP keys for future use.

It is advisable to use more than one device and replicate the private key to all of them so that there is still a copy of the private key available in case the main device is lost.

Other backup strategies are work-in-progress.

Journal

The journal consists of a json file with entries for distinct operations:

  • Incoming messages
  • Outgoing messages
  • Key exchange operations
  • IMDN notifications
  • Message removal
  • Conversation read
  • Conversation removal

Check the source code of the sample clients for working examples of parsing the journal.

Not all primitives are available for SIP classic end-points.

Setup

Sylk Server

Install Sylk Server and enable webrtc_gateway application. Initialize the storage.

There is nothing else to configure (zero configuration).

Database management

Using provided sylk-db maintenance tool one can see details about the storage usage or show information about individual accounts.

agp@sylkserver-test:~$sylk-db show test@sylk.link 
  ...... Reading configuration from /etc/sylkserver/config.ini
  ...... 
  ...... ************************ SylkServer - Show account data ************************
  ...... 
  ...... Reading storage configuration from /etc/sylkserver/webrtcgateway.ini
  ...... New Cassandra host <Host: 10.0.0.146:9042 GlobalSwitch> discovered
  ...... Server has keyspace sipthor with replication strategy: NetworkTopologyStrategy
  ...... 
  ...... --------------------------------------------------------------------------------
  ......                                  test@sylk.link                                 
  ...... --------------------------------------------------------------------------------
  ...... Message storage is enabled
  ...... Last login at:  2021-11-12 12:08:45.356000
  ...... 
  ...... 459 messages stored
  ...... 
  ......          Text Messages: 134
  ......          IMDN messages: 308
  ......         Other Messages: 17
  ...... 
  ......   Unread text Messages: 1
  ...... 
  ...... 1 push token(s) stored
  ...... 
  ......               App: com.agprojects.sylk-ios.prod
  ......         Device ID: 2EBAAC58-D245-48EE-82E6-53FD4EF0D57
  ......             Token: cac10301b55661d46b3fbb ...
  ...... 
  ...... 1 public key(s) stored
  ...... 
  ......         -----BEGIN PGP PUBLIC KEY BLOCK-----
  ......         Version: fast-openpgp
  ......         
  ......         xsFABGExS1wBEAC660kacrxXINz/1DtZ94p0mJuChgnkKShg0n6cQDbSN7vh2/hs
  ......         TBYf94eQkYT2gU0ZRBcsJinC6opnQHLTEk3grBHE2fJGU6czoEpej1HZHegVa/D
  ......         orfZieBFmmH4Sqk8j6I6mNg+UQ2RzI8WAUcXv0hOc9OQzZw3NhoDkDJGXF9bsNTk
  ......         QoHjlVxX7XFJQui8AecMDrD/h11iF01XuwTOchDCcU5waTlXztyNn2FJVufgX1gd
  ......         B+kWgOMFu0VWC++N3132KwA4uJb3eoaocZEmRKEM
  ......         =1sgu
  ......         -----END PGP PUBLIC KEY BLOCK-----

OpenSIPS

Sylk Server relies on a SIP Proxy for dispatching its messages.

Bellow is the logic code for OpenSIP that implements forking of messages to Sylk Server:

if ($rm=="MESSAGE") {
    if (db_does_uri_exist($ru, "subscriber")) {
        t_on_failure("FAILED_MESSAGE");
    }

    $avp(sip_application_type) = "message";
    $avp(can_uri) = $tu;
    $avp(source_ip) = $si;
    $avp(source_port) = $sp;
    $avp(sip_proxy_ip) = "192.168.1.100";

    if (not has_totag()) {
        t_on_reply("DIALOG_REPLY");
        if (loose_route()) {
            xlog("L_WARN", "[CONFIG] WARNING: Incorrectly formatted $rm request. Rejected with 400. ($ci)\n");
            sl_send_reply(400, "Incorrectly formatted request");
            return;
        }

        if (is_from_local()) {
            if (isflagset("DO_AUTHENTICATE_FLAG")) {
                if (not proxy_authorize("", "subscriber")) {
                    xlog("L_INFO", "[CONFIG] Asking for $fU@$fd credentials ($ci)\n");
                    proxy_challenge("", "auth,auth-int");
                    return;
                } else if ($au != $fU) {
                    xlog("L_INFO", "[CONFIG] Rejected with 403 because $au != $fU ($ci)\n");
                    sl_send_reply(403, "Username!=From not allowed ($au!=$fU)");
                    return;
                }

                # Hide auth credentials to downstream routers
                consume_credentials();
            }
        }

        if ($avp(source_ip) != "10.0.0.1") {
            # Inform Sylk to replicate an outgoing message
            append_hf("X-Sylk-From-Sip: yes\r\n");
        }

        if (is_uri_host_local()) {
            if (db_does_uri_exist($ru, "subscriber")) {
                if ($hdr(Content-Type) == "application/sylk-api-token") {
                    # api-token requests from ourselves must be only sent to Sylk Server
                    if ($avp(source_ip) != "10.0.0.1") {
                        xlog("L_INFO", "[CONFIG] Route $rm $hdr(Content-Type) $ru to Sylk Server 10.0.0.1\n");
                        $du = "10.0.0.1";

                        if (not t_relay()) {
                            sl_reply_error();
                        }
                        exit;
                    } 
                    # replies from Sylk Server with token will be routed only to the end-points
                } else if (is_present_hf("X-Replicated-Message")) {
                    xlog("L_DBG", "[CONFIG] Skip forking $hdr(Content-Type) X-Replicated-Message MESSAGE to Sylk server ($ci)\n");
                    exit;
                }
                # Should we fork message to ourselves to have a copy on other online devices 
                $var(must_fork) = 1;

                if ($var(must_fork) && $hdr(Content-Type) == "application/im-iscomposing+xml") {
                    $var(must_fork) = 0;
                }
                if ($var(must_fork) && search_body("message/imdn+xml")) {
                    $var(must_fork) = 0;
                }
                if ($var(must_fork) && $ru != 'sip:$fu@$fd') {
                    $var(must_fork) = 0;
                }
                if ($var(must_fork) && is_present_hf("X-Replicated-Message")) {
                    $var(must_fork) = 0;
                } 
                $var(orig_ru) = $ru;
                if (lookup("location")) {
                    append_branch();
                    xlog("L_INFO", "[CONFIG] Fork $rm $hdr(Content-Type) $ru to Sylk Server 10.0.0.1\n");
                    seturi("10.0.0.1");
                    append_hf("X-Sylk-App: webrtcgateway\r\n");

                    if ($var(must_fork)) {
                        $var(fu) = "sip:" + $fU + "@" + $fd;
                        xlog("L_INFO", "[CONFIG] Forking MESSAGE for $var(orig_ru) to myself $var(fu) ($ci)\n");
                        append_branch();
                        seturi($var(fu));
                        append_hf("X-Replicated-Message: yes\r\n");
                    }
                    if (not t_relay()) {
                        sl_reply_error();
                    }
                    exit;
                } else {
                    xlog("L_INFO", "[CONFIG] $ru is not yet online ($ci)\n");
                    xlog("L_INFO", "[CONFIG] Forward $rm $hdr(Content-Type) for $var(orig_ru) to Sylk Server 10.0.0.1\n");
                    $ru = "10.0.0.1";
                    append_hf("X-Sylk-App: webrtcgateway\r\n");
                    if ($var(must_fork)) {
                        $var(fu) = "sip:" + $fU + "@" + $fd;
                        xlog("L_INFO", "[CONFIG] Forking MESSAGE for $var(orig_ru) to myself $var(fu) ($ci)\n");
                        append_branch();
                        seturi($var(fu));
                        append_hf("X-Replicated-Message: yes\r\n");
                    }
                    sl_send_reply(480, "User not online");
                    return;
                }
            } else {
                sl_send_reply(404, "User not found");
                return;
            }
        }
    } else {
        # In-dialog MESSAGE
        if (not loose_route()) {
            # Only relay in-dialog requests that were previously Record-Routed by us
            sl_send_reply(400, "In-dialog MESSAGE rejected");
            exit;
        }
    }

Full instructions for installing this configuration are available at:

http://download.ag-projects.com/OpenSIPS/INSTALL

Next Steps

Continue by: