Routes: Messaging ServerAG Projects RTC Platforms (Routing Guides)
Messaging Server
Sylk Server provides online and offline messaging services for standard SIP end-points and WebRTC clients that support SylkRTC API.
The functionality is similar to what WhatsApp and Telegram offer with the difference that both client and server software are fully open source and based on open standards. Reference Web, Mobile, Desktop implementations and a live service are available for testing and interoperability purposes.
Sylk Server maintains a journal for each user with the messages and their modifications encapsulated using the following standards:
- SIP Message method - RFC3261
- CPIM Message Format - RFC3862
- Instant Message Disposition Notification (IMDN) - RFC5438
- OpenPGP - https://datatracker.ietf.org/wg/openpgp/about/
SIP offline messages can by synced between all devices by applying the journal of operations for each user.
For real time synchronization of online devices, standard SIP parallel forking is used.
The storage is transparent for end-users and it is up to the end-user devices to encrypt the payloads before sending them out by using an OpenPGP implementation.
Sylk Server has support for managing the public and private keys for PGP encrypted messages.
Features
- Message replication between multiple devices
- Offline storage when devices are not online
- Message delivery and display reports
- Message deletion, partial or per contact
- Tracks remaining unread messages
- Incremental journal per account
- Distributed storage support using Cassandra
- SIP and WebRTC APIs
- Mobile push notifications
- PGP public/private keys management
API
The functions are documented in this JavaScript API:
Sample clients
Complete fully featured clients implementing SylkRTC API:
- Blink for Mac: https://github.com/AGProjects/blink-cocoa
- Sylk Mobile: https://github.com/AGProjects/sylk-mobile
- Sylk Web and Desktop: https://github.com/AGProjects/sylk-webrtc
Content types
Sylk Server handle in special ways different content types.
application/sylk-api-token
Allows a SIP device to retrieve an authentication token that can be later used for authorization to retrieve the journal using HTTP requests.
Example:
SENDING: Packet 60, +0:01:52.503136 2021-11-15 14:33:07.194842: 192.168.1.11:50768 -(SIP over TLS)-> 85.17.186.23:5061 MESSAGE sip:ag@sip2sip.info SIP/2.0 Via: SIP/2.0/TLS 192.168.1.11:50768;rport;branch=z9hG4bKPj8MUCkGXTF7nXCTYxcMHYL5tzC8p2j-;alias From: <sip:ag@sip2sip.info>;tag=ePBmQXDzJCFfRZR3FekS3mTjCsapf To: <sip:ag@sip2sip.info> Call-ID: X6saJssTX.wmSZYDf2GrZXOAoGMiEBh0 User-Agent: Blink 8.7.0 (MacOSX) Content-Type: application/sylk-api-token Content-Length: 14 I need a token
Response:
RECEIVED: Packet 62, +0:01:52.816791
2021-11-15 14:33:07.508497: 85.17.186.23:5060 -(SIP over TCP)-> 192.168.1.11:50735
MESSAGE sip:46139205@192.168.1.11:50685;transport=tcp SIP/2.0
Via: SIP/2.0/TCP 85.17.186.23:5060;branch=z9hG4bKf3d6.33c61951.0;i=57fa5934
Via: SIP/2.0/TCP 85.17.186.26:35305;received=85.17.186.26;rport=35305;branch=z9hG4bKPj5eb777ff-0026-44b0-b17b-9d468fe76931;alias
From: "SylkServer" <sip:sylkserver@85.17.186.26>;tag=7449abda-1da-4df3-b8d1-0c94cb031f18
To: <sip:ag@sip2sip.info>
Call-ID: 3507f82d-f027-4d3a-bfb6-3efb4655caa1
User-Agent: SylkServer-6.0.0
Content-Type: application/sylk-api-token
Content-Length: 153
{"token": "iKQI7svOtyGkfsfsfsh9qGstVUOnTgrFxpo1ik40", "url": "https://webrtc-gateway.sipthor.net:9943/webrtcgateway/messages/history/ag@sip2sip.info"}By sending a GET to the URL, one can retrieve the journal since the last id.
if last_id:
url = "%s/%s" % (url, account.sms.history_last_id)
req = urllib.request.Request(url, method="GET")
req.add_header('Authorization', 'Apikey %s' % account.sms.history_token)Typically the request is sent every time the device connects to the network and then saves the last id that can be used later to resume the journal processing.
Sample code:
https://github.com/AGProjects/blink-cocoa/blob/master/SMSWindowManager.py
text/pgp-public-key
Once a user sends out a message with such payload, it will be stored in the user profile and end-points that implement public key fetch function can retrieve the key before starting a conversation.
Sample code:
https://github.com/AGProjects/sylkrtc.js/blob/master/lib/account.js
text/pgp-private-key
This is used for end-points to replicate their private keys, so multiple devices can by synchronized.
To replicate messages on multiple devices you need the same private key on all of them.
To be compatible with existing clients the message must be addressed to the same user and the payload must consist of the user PGP public key in clear text followed by the PGP encrypted with a symmetric code private key.
Example:
SENDING: Packet 88, +0:35:02.954481 2021-11-15 15:06:17.646187: 192.168.1.11:51541 -(SIP over TLS)-> 85.17.186.23:5061 MESSAGE sip:ubuntu@test.com SIP/2.0 Via: SIP/2.0/TLS 192.168.1.11:51541;rport;branch=z9Xb--OH3b91;alias Max-Forwards: 70 From: <sip:ubuntu@test.com>;tag=pPOSjBJx2nm9ZG8DZX-ObDtCfDaqJBDk To: <sip:ubuntu@test.com> Call-ID: bhNsVTpEoFNNqXtNcpm CSeq: 52490 MESSAGE User-Agent: Blink 8.7.0 (MacOSX) Content-Type: text/pgp-private-key Content-Length: 5378 -----BEGIN PGP PUBLIC KEY BLOCK----- xsFNBGDQz0cBEACw7ZCd2CIw5udWY5VOV4ZrzvyIdt8idBoUqbUJ6Lm55KVMj7Kh 6Lu13BKfV3m04Q3L3b5KboqIsbJLivlpg2YX+vOAAzEDE9BTZN0o+orDWxP1Dz9U LvtloSRoNK1NZj+KkbKfRXUZCMnyXDiN64cYSFLGp0esT/DhyfI09p63dqegl+Ut uenVLzne9PkpdewS0OUmfKUGBY86SPUewwBkFFVxYyIWQu+yJLfMt7jxTHnD3dcT mDaRljMZj6qhzQ2F2sc68KHy2FIeXR9kOxMq8T6aEw0JwAA4rLWWCT6MLEtjDPng cduwml6F7g== =Xp4B -----END PGP PUBLIC KEY BLOCK----- -----BEGIN PGP MESSAGE----- O+rMXHZq7rhMlYyBYJFSgVZTn2t58s/3NkTVewwvgstl10BUDgn9zBggElo86duB 2fqwEGCYSny1CLfWtMe/8GBrLd/pCpAWVlzsaI/s/7QbLqW8d0nq1uft8ubRvyWb wIl0lLwuSReeyNcFHw53jpumFQbXYtZRM8UhhKmJoeKwZh++AIYUdG5OlPUb32hF 8mhmr1m1rhxPJTWQvaCsYg== =sZwQ -----END PGP MESSAGE-----
The receiving end-point must decrypt the message and save the PGP keys for future use.
It is advisable to use more than one device and replicate the private key to all of them so that there is still a copy of the private key available in case the main device is lost.
Journal
The journal consists of a json file with entries for distinct operations:
- Incoming messages
- Outgoing messages
- Key exchange operations
- IMDN notifications
- Message removal
- Conversation read
- Conversation removal
Check the source code of the sample clients for working examples of parsing the journal.
Setup
Sylk Server
Install Sylk Server and enable webrtc_gateway application. Initialize the storage.
There is nothing else to configure (zero configuration).
Database management
Using provided sylk-db maintenance tool one can see details about the storage usage or show information about individual accounts.
agp@sylkserver-test:~$sylk-db show test@sylk.link ...... Reading configuration from /etc/sylkserver/config.ini ...... ...... ************************ SylkServer - Show account data ************************ ...... ...... Reading storage configuration from /etc/sylkserver/webrtcgateway.ini ...... New Cassandra host <Host: 10.0.0.146:9042 GlobalSwitch> discovered ...... Server has keyspace sipthor with replication strategy: NetworkTopologyStrategy ...... ...... -------------------------------------------------------------------------------- ...... test@sylk.link ...... -------------------------------------------------------------------------------- ...... Message storage is enabled ...... Last login at: 2021-11-12 12:08:45.356000 ...... ...... 459 messages stored ...... ...... Text Messages: 134 ...... IMDN messages: 308 ...... Other Messages: 17 ...... ...... Unread text Messages: 1 ...... ...... 1 push token(s) stored ...... ...... App: com.agprojects.sylk-ios.prod ...... Device ID: 2EBAAC58-D245-48EE-82E6-53FD4EF0D57 ...... Token: cac10301b55661d46b3fbb ... ...... ...... 1 public key(s) stored ...... ...... -----BEGIN PGP PUBLIC KEY BLOCK----- ...... Version: fast-openpgp ...... ...... xsFABGExS1wBEAC660kacrxXINz/1DtZ94p0mJuChgnkKShg0n6cQDbSN7vh2/hs ...... TBYf94eQkYT2gU0ZRBcsJinC6opnQHLTEk3grBHE2fJGU6czoEpej1HZHegVa/D ...... orfZieBFmmH4Sqk8j6I6mNg+UQ2RzI8WAUcXv0hOc9OQzZw3NhoDkDJGXF9bsNTk ...... QoHjlVxX7XFJQui8AecMDrD/h11iF01XuwTOchDCcU5waTlXztyNn2FJVufgX1gd ...... B+kWgOMFu0VWC++N3132KwA4uJb3eoaocZEmRKEM ...... =1sgu ...... -----END PGP PUBLIC KEY BLOCK-----
OpenSIPS
Sylk Server relies on a SIP Proxy for dispatching its messages.
Bellow is the logic code for OpenSIP that implements forking of messages to Sylk Server:
if ($rm=="MESSAGE") {
if (db_does_uri_exist($ru, "subscriber")) {
t_on_failure("FAILED_MESSAGE");
}
$avp(sip_application_type) = "message";
$avp(can_uri) = $tu;
$avp(source_ip) = $si;
$avp(source_port) = $sp;
$avp(sip_proxy_ip) = "192.168.1.100";
if (not has_totag()) {
t_on_reply("DIALOG_REPLY");
if (loose_route()) {
xlog("L_WARN", "[CONFIG] WARNING: Incorrectly formatted $rm request. Rejected with 400. ($ci)\n");
sl_send_reply(400, "Incorrectly formatted request");
return;
}
if (is_from_local()) {
if (isflagset("DO_AUTHENTICATE_FLAG")) {
if (not proxy_authorize("", "subscriber")) {
xlog("L_INFO", "[CONFIG] Asking for $fU@$fd credentials ($ci)\n");
proxy_challenge("", "auth,auth-int");
return;
} else if ($au != $fU) {
xlog("L_INFO", "[CONFIG] Rejected with 403 because $au != $fU ($ci)\n");
sl_send_reply(403, "Username!=From not allowed ($au!=$fU)");
return;
}
# Hide auth credentials to downstream routers
consume_credentials();
}
}
if ($avp(source_ip) != "10.0.0.1") {
# Inform Sylk to replicate an outgoing message
append_hf("X-Sylk-From-Sip: yes\r\n");
}
if (is_uri_host_local()) {
if (db_does_uri_exist($ru, "subscriber")) {
if ($hdr(Content-Type) == "application/sylk-api-token") {
# api-token requests from ourselves must be only sent to Sylk Server
if ($avp(source_ip) != "10.0.0.1") {
xlog("L_INFO", "[CONFIG] Route $rm $hdr(Content-Type) $ru to Sylk Server 10.0.0.1\n");
$du = "10.0.0.1";
if (not t_relay()) {
sl_reply_error();
}
exit;
}
# replies from Sylk Server with token will be routed only to the end-points
} else if (is_present_hf("X-Replicated-Message")) {
xlog("L_DBG", "[CONFIG] Skip forking $hdr(Content-Type) X-Replicated-Message MESSAGE to Sylk server ($ci)\n");
exit;
}
# Should we fork message to ourselves to have a copy on other online devices
$var(must_fork) = 1;
if ($var(must_fork) && $hdr(Content-Type) == "application/im-iscomposing+xml") {
$var(must_fork) = 0;
}
if ($var(must_fork) && search_body("message/imdn+xml")) {
$var(must_fork) = 0;
}
if ($var(must_fork) && $ru != 'sip:$fu@$fd') {
$var(must_fork) = 0;
}
if ($var(must_fork) && is_present_hf("X-Replicated-Message")) {
$var(must_fork) = 0;
}
$var(orig_ru) = $ru;
if (lookup("location")) {
append_branch();
xlog("L_INFO", "[CONFIG] Fork $rm $hdr(Content-Type) $ru to Sylk Server 10.0.0.1\n");
seturi("10.0.0.1");
append_hf("X-Sylk-App: webrtcgateway\r\n");
if ($var(must_fork)) {
$var(fu) = "sip:" + $fU + "@" + $fd;
xlog("L_INFO", "[CONFIG] Forking MESSAGE for $var(orig_ru) to myself $var(fu) ($ci)\n");
append_branch();
seturi($var(fu));
append_hf("X-Replicated-Message: yes\r\n");
}
if (not t_relay()) {
sl_reply_error();
}
exit;
} else {
xlog("L_INFO", "[CONFIG] $ru is not yet online ($ci)\n");
xlog("L_INFO", "[CONFIG] Forward $rm $hdr(Content-Type) for $var(orig_ru) to Sylk Server 10.0.0.1\n");
$ru = "10.0.0.1";
append_hf("X-Sylk-App: webrtcgateway\r\n");
if ($var(must_fork)) {
$var(fu) = "sip:" + $fU + "@" + $fd;
xlog("L_INFO", "[CONFIG] Forking MESSAGE for $var(orig_ru) to myself $var(fu) ($ci)\n");
append_branch();
seturi($var(fu));
append_hf("X-Replicated-Message: yes\r\n");
}
sl_send_reply(480, "User not online");
return;
}
} else {
sl_send_reply(404, "User not found");
return;
}
}
} else {
# In-dialog MESSAGE
if (not loose_route()) {
# Only relay in-dialog requests that were previously Record-Routed by us
sl_send_reply(400, "In-dialog MESSAGE rejected");
exit;
}
}Full instructions for installing this configuration are available at:
http://download.ag-projects.com/OpenSIPS/INSTALL
Next Steps
Continue by:
- learning about Routes: End-Point; or
- learning about Routes: MSRP media; or
- learning about Routes: Presence; or
- learning about Routes: PSTN; or
- learning about Routes: Special; or
- going back to the Overview.