Page Menu
Home
Phabricator
Search
Configure Global Search
Log In
Files
F7159972
RATING.txt
No One
Temporary
Actions
View File
Edit File
Delete File
View Transforms
Subscribe
Mute Notifications
Award Token
Flag For Later
Size
30 KB
Referenced Files
None
Subscribers
None
RATING.txt
View Options
CDRTool rating engine
---------------------
CDRTool provides on-the-fly rating of CDRs from multiple data-sources like
OpenSIPS, Asterisk or Cisco gateways based on easy to build rating plan,
which can be provisioned in a web interface or imported from csv files.
A Call Detail Record (CDR) is one record from the radius radacct table. The
CDR contains all information related to a session, its duration, the calling
and called party and media information. The rating engine calculates the
price of the session. The calculation is done once and its results are saved
in the CDR table for later use. It is possible to re-calculate the prices
at a later time for example when having to change the tariffs.
Based on exceptions, different rates may be applied per caller party
identified by source IP, domain or subscriber.
Rating is applied only after the call has ended. CDRTool considers that a
call has ended when there is a stop time.
In case of missing BYEs, CDRTool relies on the fact that MediaProxy will
update the CDR with the proper stop time information based on the last
moment the media stream passed through the media proxy. This functionality
is not available when the end-points have negotiated using ICE another
candidate than MediaProxy.
The rates are linked with profiles corresponding with different time of the
day, day of the week or holidays. For rating calls, which span multiple
profiles, the right rate is selected and applied for the call duration
within each profile. Each customer may be assigned its own dedicated rating
plans destination id and names. Chained profiles are possible to enable
exception based rating. Multiple customers may share a common rate list,
while some destinations may be rated differently, only the differences must
be provisioned. Multiple time zones are supported for multiple billing
parties hosted on the same platform.
Rating logic
------------
Different subscribers can have different rating plans. A rating plan is a
unique combination of holidays, day of week, time of day, destination ids,
and associated costs.
The following steps are performed to rate a CDR:
1. Determination of the billing party
2. Determination of the destination id
3. Determination of the costs
The steps are described in detail below.
1. Determination of the billing party
-------------------------------------
To be able to calculate the Price for a call the rating engine must
determine whose rating plan to use.
The rating engine does this by performing a match against entries in the
billing_customers table for the the BillingPartyId field of the the CDR
(radacct.UserName for radius based datasources) in the following order:
a. SIP account user@domain
b. SIP domain of the SIP account
c. Source IP of the session
d. Default (when none of the above matches)
You must have a maximum of one column (either subscriber, domain or gateway)
set in every row of billing_customers table for this to work.
The first match is considered to be the billing party for which the rating
plan is determined.
The rating plan is further derived from the profiles associated with the
entry found in the billing_customers table.
Note
The billing_customers table field that matches the Source IP is called
gateway. "gateway" or "trusted peer" terms are used interchangeably in this
document. They both relate to the source IP address that generated the SIP
session.
If you want to use the multi-tenant features for OpenSIPS accounting you
must create an integer reseller_id column in the trusted and domain tables.
Traffic generated by IP addresses from the trusted peers table and local
domains will be marked as belonging to their correspondent reseller_id. The
reseller_id is then later used to lookup destinations and rates belonging to
the reseller_id.
2. Determination of the destination id
--------------------------------------
The rating engine identifies the 'destination id', which has associated
rates depending on day of week and time of day.
The 'destination id' is for example a country prefix like '31' for the
Netherlands. This 'destination id' has prices associated with it so all
calls to the Netherlands will have prices associated with prefix '31'.
The 'destination id' is derived from the logical destination the SIP session
has been routed to. In SIP headers and Radius records there are multiple
places that contain information related to the destination. Some of them are
generated by the SIP User-Agents (hence cannot be trusted and must not be
used for accounting purposes) and others are generated by the SIP Proxy
configured by the operator and are suitable for accounting purposes.
The rating engine considers the destination to be the first non-empty CDR
field in this order:
a. CanonicalURI (the destination after all lookups inside the SIP Proxy)
b. SipTranslatedRequestURI (the Request URI as presented by the SIP UA)
c. CalledStationId (the content of the To header, used as a last resort)
The CanonicalURI is the preferred because is reliable information generated
by the operator based on the logic configured in the SIP Proxy and the
subscriber cannot control it. OpenSIPS must be configured to send the
CanonicalURI Radius attribute when creating the radius accounting START
record. Instructions for how to set this up are available in INSTALL.txt
file.
The 'destination id' is then calculated based on the longest match of this
destination field in the billing_destinations table.
The longest match for the chosen destination field is performed by the
E164_class, which by default uses E164_Europe that defines an European
numbering plan. It assumes the destinations start with a zero for a national
call and with double zero for an international call. See cdr_generic.php for
the actual logic.
If your dialing plan is different, you must use other provided class like
E164_US or create a custom class and point to it in global.inc for each
datasource as follows:
'E164_class'=>'MyE164Class',
For example the pre-defined E164_US class from cdr_generic.php matches the
American dialing plan.
Destinations are uploaded into destination table. Each reseller_id can have
its own list of destinations. If the reseller_id of trusted peers and domain
tables have been set, then the rating engine will apply the rating
information belonging to them.
Each destination has also a set of properties that can be used to alter the
calculation of the price. These properties are:
- increment: the call duration is rounded to the next increment in seconds,
this is only used for rating, the actual CDR duration is not modified
- min_duration: set the duration to min_duration if the CDR duration is
smaller than min_duration, this is only used for rating, the actual CDR
duration is not modified
- max_duration: limit the duration to max_duration if CDR duration is
bigger than max_duration, this is only used for rating, the actual CDR
duration is not modified
- max_price: the price is set to max_price per call if the calculated price
is bigger than the max_price
The min_duration and increment values from billing_customers override the
values found in the destinations table.
3. Determination of the costs
-----------------------------
The following steps are taken to determine the cost for the calls based on
the 'destination id' and the billing party determined at the previous steps.
1. Lookup the billing profiles in cdrtool.billing_customers table in the
following order: subscriber,domain,gateway (based on $this->dayofweek):
(as explained in "Determination of the billing party" above)
- profile_name1 matches week days [1-5]
- profile_name1_alt matches week days [1-5] if no rates for profile_name1
are found
- profile_name2 matches week-ends [6-0]
- profile_name2_alt matches week-ends [6-0] if no rates for profile_name2
are found
- profile_name2 matches also holidays from billing_holidays table
The week starts with 0 (Sunday) and ends with 6 (Saturday)
This step determines which rates should be applied based on the day of
the week when the call started.
2. Using the profile_name found, lookup the rate_name based
on $this->hourofday in cdrtool.billing_profiles table
If no rate_name is found for the given profiles a second set of profiles
are used, profile_name1_alt and profile_name2_alt.
- the day may be split in maximum 4 periods
- the days starts with hour 0 and ends with hour 24
- rate_name1 defines the first interval after hour 0
- rate_name2 defines the first interval after rate_name1
- rate_name3 defines the first interval after rate_name2
- rate_name4 defines the first interval after rate_name3
When the hour matches an interval use the rate_nameX found to lookup the
rate in billing_rates, if no record is found use the rate called
'default'
This step determines which rate should be applied for the time of day
when the call started.
3. Lookup in the cdrtool.billing_rates table the record having same name
found at point 2 having billing_rates.destination = 'destination id'
and billing_rates.application = application type found in the steps above.
- return an array with all the rating values and the duration rated
No rate will be returned if no 'destination id' is found. Make sure each
possible destination has a 'corresponding id' and name in the destinations
table.
The 'audio' application may optionally be specified with a subtype in order
to refine billing rates. If a subtype is specified, it is appended to the
application name, separated by a dot (i.e. 'audio.incoming').
For example, ('1', 'audio') can have a durationRate of 280, where
('1', 'audio.incoming') can have a durationRate of 0. If the application
isn't specified in the CDR, it defaults to 'audio'.
This step determines the costs within the current time span associated
with the time of day and destination id. If the call duration exceeds
this time span (that is a new interval for which another rate applies is
reached), step 4 is performed.
4. If the duration rated at point 3 is less than total call duration, apply
point 3 again for the remaining call duration in the next profile. A
maximum of 10 spans (different rates depending of time of day, day of the
week) can be calculated using this mechanism. After 10 spans, the engine
bails out to avoid loops caused by invalid tables provisioning.
5. Calculate the total call Price based on its duration and connection fees.
In global.inc there are several variables that affect how the price is
calculated. These settings are global per CDRTool installation but some
can be overwritten with per customer values in the billing_customers
table.
$RatingEngine=array(
"priceDenominator" => 10000, // Rates units (global setting)
"priceDecimalDigits" => 4, // Decimal information (global setting)
"minimumDurationCharged" => 0, // Rate a minimum of X seconds (per customer)
"minimumDuration" => 0, // Minimum duration to rate, if call duration is shorter the price is zero (per customer)
"durationPeriodRated" => 60 // Rate is per 60 seconds (global setting)
"trafficSizeRated" => 1024, // Default we rate per 1 MB (global setting)
"reportMissingRates" => 0 // Send emails to administrator in case of missing rates
);
Pricing formula
---------------
if min_duration then
minimumDurationCharged = min_duration
else if minimumDurationCharged set in global inc
use minimumDurationCharged from global.inc
else
minimumDurationCharged = call duration
if increment then
durationForRating = round to the next increment
else
durationForRating = call duration
if durationForRating >= minimumDurationCharged then
Price = connectCost/priceDenominator+
durationRate*durationForRating/durationPeriodRated/priceDenominator
else
Price = 0
ENUM discounts
--------------
The rating engine can apply a discount associated with the ENUM top
level domain that returned the final destination.
Price = Price - Price * ENUM discount / 100
To apply ENUM based discounts, the ENUM TLD must be saved with each CDR
and the TLDs with their corespondent discounts must be provisioned in the
Rating tables section. See ENUM TLD discounts section for more
information.
Purchasing price
----------------
A second price called 'Price in' is calculated using the same formula but
based on connectCostIn and durationRateIn values. It can be used to match
the purchasing price and calculate the margin between purchasing and
selling prices. The information about both prices is stored in the
RateInfo field of the CDR. The values for connectCostIn and
durationRateIn must be provisioned in the billing_rates and
billing_rates_history tables using the web interface or by importing csv
files.
6. Save the calculated Price, billing party and 'destination id' for each
call in the CDR table. Having the price stored in the database, it is
possible to build statistics to display consolidated revenues per
destination id, gateway, domain or subscriber.
Testing the rating engine
-------------------------
You can test the rating engine by telneting to the IP and port configured in
global.inc. Type 'help' once connected to see the available commands. Use
'ShowPrice' command to simulate the rating of one session, for example:
adigeo@w1:/var/www/CDRTool/doc$ telnet ws1 9024
Trying 10.0.0.1 ...
Connected to 10.0.0.1.
Escape character is '^]'.
ShowPrice From=sip:123@example.com To=sip:0031650222333@example.com Gateway=10.0.0.1 Duration=59
0.2023
Duration: 59 s
App: audio
Destination: 31650
Customer: domain=example.com
Connect: 0.0450
StartTime: 2009-01-03 14:29:10
--
Span: 1
Duration: 59 s
ProfileId: 442 / weekend
RateId: 442 / 0-24h
Rate: 0.1600 / 60 s
Price: 0.1573
Below is a description of the fields that must be separated by one or more
spaces:
* From - must contain the caller party
* Gateway - must contain the source IP of the session
* To - must contain the full CanonicalURI destination, its format must be synced
with the E164 class logic used to determine the 'destination id'
* Duration - the duration of the session in seconds
Check the syslog for any errors, most of the configuration errors like
missing rates are logged to the syslog.
Importing and exporting of rating files
---------------------------------------
It is important to set the impersonate field correctly for the login
accounts. To access in the CDRTool Rating web page all the rates table for
all resellers in the system you must set the impersonate field of the login
account to 0.0
There are different data files needed for rating. The data files are
imported into their corresponding MySQL tables.
The files must be uploaded to /var/spool/cdrtool directory. To load the
files into the database run the following command:
/var/www/CDRTool/scripts/importRatingTables.php
Data partitioning for multiple resellers
The reseller_id column present in all rating tables is used to filter access
based on Login account impersonate field.
You may create numerical sub-directories under /var/spool/cdrtool directory.
When importing a file from such numerical directory, the reseller_id field
in all imported records is set to the directory name.
The import script knows to import the files only once so you may dump
several files over time with the same name and safely run the import script
from cron. The import script detects whether each file has already been
imported by building a unique key out of the filename and the hash of the
file content. So you may use the same filenames as long as the content
differs and viceversa. If the import file has changed any records, the
rating engine is automatically instructed to reload the changes.
Sample csv files are found in the setup directory. The CSV field order is
described in setup/*.csv sample files. The first element on each line
specifies the operation will be performed with the current record. The
operation can be 2 (update/insert), 1 (insert) or 3 (delete).
The updates are performed based on a unique key present in each table:
billing_customers - cust_idx (reseller,gateway,domain,subscriber)
destinations - cust_dest_idx (reseller,gateway,domain,subscriber,dest_id)
billing_profiles - profile_idx (reseller,name)
billing_rates - rate_idx (reseller,name,destination,application)
billing_rates_history - rate_idx (reseller,name,destination,application,startDate,endDate)
The content of the rating tables can be exported in the Rating tables page.
The import script detects the type of file to import based on its filename.
The filename must comply with the following naming convention:
1. Must start with the name of the table without the billing_ prefix
2. May optionally contain extra characters after the name
3. Must end with .csv extension
Examples:
- rates.csv or rates20061201.cvs will be loaded into the rates table
- profiles.csv or profiles20061201.cvs will be loaded into the profiles table
- destinations200601.csv will be loaded in the destinations table
- ratesHistory200801.csv will be loaded in the rates_history table
Do not use 'billing_' prefix in front of the file name.
It is advisable to name the files in a consistent manner like tableYYYYMMDD.csv
The results of the import operation is logged in the database and can be
viewed in the Log section of the web interface and the syslog.
MySQL schema
------------
To see the rating tables and their structures connect to the cdrtool
database using mysql client.
Run 'show tables' and 'describe table_name':
billing_customers
+-------------------+
| Field |
+-------------------+
| id |
| reseller_id |
| gateway |
| domain |
| subscriber |
| profile_name1 |
| profile_name1_alt |
| profile_name2 |
| profile_name2_alt |
| timezone |
| increment |
| min_duration |
+-------------------+
billing_profiles
+------------+
| Field |
+------------+
| id |
| reseller_id|
| name |
| rate_name1 |
| hour1 |
| rate_name2 |
| hour2 |
| rate_name3 |
| hour3 |
| rate_name4 |
| hour4 |
+------------+
billing_rates
+-----------------+
| Field |
+-----------------+
| id |
| reseller_id |
| name |
| destination |
| application |
| connectCost |
| durationRate |
| connectCostIn |
| durationRateIn |
+-----------------+
billing_rates_history
+-----------------+
| Field |
+-----------------+
| id |
| reseller_id |
| name |
| destination |
| application |
| connectCost |
| durationRate |
| connectCostIn |
| durationRateIn |
| startDate |
| endDate |
+-----------------+
destinations
+--------------+
| Field |
+--------------+
| id |
| reseller_id |
| gateway |
| domain |
| subscriber |
| dest_id |
| region |
| dest_name |
| increment |
| min_duration |
| max_duration |
| max_price |
+--------------+
billing_holidays
+-------+
| Field |
+-------+
| day |
+-------+
prepaid
+------------------+
| Field |
+------------------+
| id |
| reseller_id |
| account |
| domain |
| balance |
| change_date |
| active_sessions |
| session_counter |
+------------------+
prepaid_history
+-------------+
| Field |
+-------------+
| id |
| reseller_id |
| username |
| domain |
| action |
| number |
| value |
| balance |
| date |
+-------------+
billing_enum_tlds
+-------------+
| Field |
+-------------+
| id |
| reseller_id |
| gateway |
| domain |
| subscriber |
| enum_tld |
| e164_regexp |
| discount |
+-------------+
Web based rating tables management
----------------------------------
The rating tables can be edited from the web, click on Rating tables link.
One may insert/update/delete records or apply the changes on selections. For
example it is possible to increase with XX units the rate for a specific
destination.
Numeric fields support mathematical operators [+-*/], one may update
using absolute or relative values the fields in the rating tables.
The rates may be copied in bulk and start quickly working with a fresh
new rating table. Select in the Rates PSTN table by filtering on rate
name. A new button appears which allows the copy of all selected rates
into a new set. The rates are copied under the old rate id with _N
suffix where N is the next available number for which same rate id does
not exist.
The content of the rating tables can be exported into comma separated
files. The CSV format has the same structure as the import file, is
fairly easy to modify an exported batch file into an external
application and load it back into CDRTool.
Note
When $RatingEngine['split_rating_table'] is true, after changing the rates
in the web interface or by importing them, you must run the script
scripts/splitRatingTables.php to split the central billing_rates table into
individual tables for each rate id. You do not need this feature if you have
less than 100K rates in your system.
ENUM TLD discounts
------------------
To apply discounts based on ENUM certain conditions must be met.
1. The username part of the result of the ENUM lookup must be numeric and
contain a fully qualified E164 number, optional with a numeric prefix.
Example:
The user dialed 020800001, the SIP Proxy has normalized the destination
based on local policy by stripping 0 and adding country code 31 to obtain
the fully qualified E164 number 3120800001, than it performed an ENUM
lookup under top level domain e164.example.com for +3120800001. The ENUM
server responsable for e164.example.com returned a response with the
destination sip:01131208000011@gateway.example.com
In the CDR, ENUMtld is stored as e164.example.com
The rating logic checks if the TLD exists in the billing_enum_tlds table.
If it does, the rating engine tries to match the regexp field against the
username part of the destination from the ENUM response, which has been
saved in the Canonical URI. The match must return a fully qualified E164
number otherwise the call is considered to have ended to a no E164
destination and the call will be free of charge.
2. In the ENUM tld table you must provision (for the example above):
- TLD: e164.example.com
- Regexp: 011([1-9][0-9]{7,})
- Discount: 25
The parenthesis of the Regexp field indicate the E164 number returned by
the match and discount is a percentage that will be subtracted form the
total price of the call. The formula is described in the PSTN rating
section.
Reloading rating tables
-----------------------
The rating engine loads some of the rating tables in the memory, when the
tables change a reload is needed. Reload of rating tables is possible
without stopping the daemon by connecting to it and issuing the reload
command. The init.d script can also be used for reloading the rating engine
with the current values from the rating database.
There are 3 ways of reloading the rating tables:
a. Each change executed in the WEB interface for rate management may update
the rating tables. If there is a change made to the database that requires a
reload the link 'Reload rating tables' appears in red color on web page.
Click on the link to execute the reload.
b. Telnet to the IP address and port number specified for the Rating engine
in global.inc. Type help to see the list of commands available. Locate the
reload rates command and execute it followed by \n. You may see the result
of the command in syslog. The results displayed by syslog will show how many
entries have been reloaded from the rating table.
c. Run /etc/init.d/cdrtool reload command
Troubleshooting
---------------
To examine the rate information for a rated call click on the Id field on
the leftmost column. (Java script support in browser is required). A blue
area will open under the CDR line containing more information about the SIP
session.
If you see no price in the CDR or no rating information appears in the call
details it means that either no destination was found in the destinations
table or no rate has been associated with that destination. Make sure that
for each entry in the destinations table there is a corespondent entry in
the rates table. CDRTool rating engine can send warning emails if it finds
missing entries in the rating tables if the system where CDRTool runs is
properly configured to send emails and the e-mail notification addresses are
set in global.inc:
$CDRTool['provider']['toEmail'] = "support@example.com";
$CDRTool['provider']['bccEmail'] = "cdrtool@example.com";
To log to syslog about missing rates or incorrect setup of the rating tables
enable 'reportMissingRates' in $CDRTool['rating'] section of global.inc
Renormalizing CDRs and historical rating
----------------------------------------
Sometime is useful to be able to change the rates for calls that have been
already normalized and rated, for example after changing the rating tables
you wish to apply the changes for the previous month for a customer.
To re-rate the CDRs do the following:
1. Change the current rates by using cvs files/WEB interface or add rates
valid for specific dates/destinations in the rates_history table
2. Re-normalize the calls to be re-rated by either selecting ReNormalize
check-box in the search screen or by changing the Normalized field in the
CDR MySQL table (e.g. radacct):
Examples:
a) Re-rate calls for this month (2004-12) SIP domain example.com:
UPDATE radacct set Normalized = '0' where Realm = 'example.com'
and AcctStartTime >= '2004-12-01'
b) Re-rate calls for SIP subscriber sip01@example.com:
UPDATE radacct set Normalized = '0' where UserName = 'sip01@example.com'
3. Apply rating again using command:
/var/www/CDRTool/scripts/normalize.php
Notes
Renormalization process can take long time during which your database
(radacct table) will be intermitently locked. Perform this operation
only during low traffic periods.
It is advisable to re-rate only the CDRs for destinations that have
different rates. To do this, select a filter in the CDR search screen, if
the selection is right re-run the query by selecting Re-normalize button.
After renormalization, the monthly usage information used by the quota
system will be out of date. At the next run of the quotaCheck script, a full
table scan will be performed. See QuotaSystem.txt for more information about
quota.
Holidays must be added as individual days YYYY-MM-DD in table
billing_holidays. The profile applied for holidays is the same as for
week-ends. Holidays are global and cannot be specified per customer.
Renormalization process does not affect the balance of prepaid users.
Prepaid is a real time un-reversible process, it goes in one direction. The
prepaid balance is changed only by placing a call or adding credit to it.
There are several reasons for this:
- The balance before and after each CDR is not known to be able to roll it
back at a later time
- Re-rating correctly is mathematically not possible for prepaid users that
have calls in progress
- If the prices are higher than previously debited and end up with a
negative balance, the software cannot force the user to pay more
retroactively
Re-normalization for the purpose of re-rating is useful only for postpaid
accounts where you send an invoice at the end of the month and your can
change things back and forth. If you need to perform manual credit/debit
operations to some prepaid users because of faulty pricing, you can edit in
CDRTool in the prepaid table the balance by using + or -.
Known limitations
-----------------
The rating engine does not calculate prices based on the outbound carriers
or outbound gateways, the rating plan is is assigned by the calling party
and not by called party.
Price discounts (except those based on ENUM tld) must be applied outside
CDRTool, in the billing system that prints the actual invoices. CDRTool has
no possibility to rate only calls after X minutes per month for subscriber
Y, all calls are rated uniformly.
Performance
-----------
Rating is part of the normalization process that happens every time a query
is executed in the web interface or when the rating engine is contacted by
the SIP Proxy or by the User Agent that performs the prepaid application.
The following tests have been performed between two machines with 3 GHz CPU
and 1 GB memory located on the same LAN having a round trip time of 0.2 ms.
The rating tables have been populated durring the tests with:
Aug 11 11:25:43 sip03 CDRTool[4945]: Loaded 8135 destinations
Aug 11 11:25:43 sip03 CDRTool[4945]: Loaded 6 profilesPSTN
Aug 11 11:25:43 sip03 CDRTool[4945]: Loaded 4 holidays
Aug 11 11:25:44 sip03 CDRTool[4945]: Loaded 7273 prepaid accounts
Tests results for the postpaid application:
Clients Rating command Execution speed
----------------------------------------------------------------
1 ShowPrice 390/s per client
5 ShowPrice 100/s per client
10 ShowPrice 60/s per client
Tests results for the prepaid application
Clients Rating command Execution speed
----------------------------------------------------------------
1 MaxSessionTime/DebitBalance 250/s per client
5 MaxSessionTime/DebitBalance 80/s per client
10 MaxSessionTime/DebitBalance 40/s per client
Client means either a SIP Proxy entity or a CDRTool server, which performs
the normalization process.
The rates are read directly from MySQL from version 6.1 instead of
being cached as they are found based on a known index.
To monitor the connections to the rating engin server telnet to
the rating engine port and issue the ShowClients command. Example output:
ShowClients
Clients:
1. 91.20.228.143:32837
2. 85.1.86.71:57945
3. 91.20.228.146:35098
4. 91.20.228.150:34285
5. 91.20.228.129:55090
6. 91.20.228.164:34147
Requests:
12 requests from 91.20.228.129
11 requests from 91.20.228.150
8 requests from 85.11.86.71
2 requests from 91.20.228.164
Statistics:
Total requests: 33
Uptime: 169 seconds
Load: 0.20/s
File Metadata
Details
Attached
Mime Type
text/plain
Expires
Sat, Nov 23, 2:19 PM (23 h, 6 m)
Storage Engine
blob
Storage Format
Raw Data
Storage Handle
3409302
Default Alt Text
RATING.txt (30 KB)
Attached To
Mode
rCDRT CDRTool
Attached
Detach File
Event Timeline
Log In to Comment