go-data-routing

The go-data-routing library provides a  #DSL for stream-oriented processing of data. Based on the concepts of #EIP (enterprise integration patterns) and concurrency primitives of #golang.

The motivation for this project is simple: to get an easy and clear way of coding ETL-like programs for parallel processing of data. In my case it was a BFS crawler tuned for extraction of specific metadata, (see a basic version in `example` folder).  

Features

The library provides the following primitives:

• route (chain of nodes processing messages)
• node:
  • filter
  • processor -- processes a stream of tasks in parallel
  • wire tap: sends a copy of msg to another route (referenced by name)
  • to: enrich msg on another route (request-reply / enrichment pattern)

All the primitives are accessible through DSL.

Design of node:

• each node is connected with the next one (if exists) only with 1 channel
• node owns an input channel
• output is just a reference to the input of next node
• node does not close the output channel, instead it just sends a Stop msg to a next node
• if a node is the last in a chain than an output message being sent is discarded unless it's not a RequestReply

go-pool-of-workers

This is a simplistic Go implementation of the pool of workers [1].

When you might need it:

If you want to handle a number/stream of jobs in parallel but with a limited number of goroutines.

All you need is:

• a job Runner -- a handler for a specific unit of work
• an optional callback to handle a result of job

Code example:

type Job struct{ result SomeType }

func (r *Job) Run() {
    time.Sleep(200 * time.Millisecond)
    r.result = x
}

func main() {
    p := pool.NewPool(2, 4) // minWorkers, maxWorkers
    p.fnOnResult = func(handledJob Job) {}

    tasks := 10
    for tasksCnt > 0; tasksCnt-- {
        p.Submit(&Job{})
    }
    p.Stop()
}

Design

The key difference from canonical pool [2] is an absence of a common job queue. Instead, there is a common queue of idle workers [1][2].

Features / properties

• The producer (submitting a job) is unlocked as soon as a idle worker consumes the job, thus potentially reducing the time producer is blocked.
• Number of workers increases on demand from minWorkers to maxWorkers
• You may have as many types of jobs as you like

References

1. Go: Worker Pool vs Pool of Workers
2. Go by Example: Worker Pools
3. Handling 1 Million Requests per Minute with Go

SMPP for Go developers and not only

FAQ about SMPP

1. Q: How to get notification on subscriber availability using HLR
   
   A:
   An ESME may use the set_dpf parameter [in the submit_sm] to request the setting of a delivery pending flag (DPF) for certain delivery failure scenarios, such as MS [i.e. Mobile Station] unavailability (as indicated by the HLR). The MC [i.e. Message Center] should respond to such a request with an alert_notification PDU when it detects that the destination MS has become available. For more information see https://www.activexperts.com/sms-component/sms/smpptlv/
2. Q: Каковы преимущества использования SMPP
   
   A: 
- стандартный протокол (не придется переписывать бек-энд при смене SMS-шлюза)
- асинхронность, высокая пропускная способность
- отправка сообщений происходит практически мгновенно
- оперативная доставка статуса доставки и ошибок и бОльшая информативность,
точность времени доставки до минуты
- возможность проверка номера и дислокации абонента при помощи HLR-запроса (напр. есть такие ошибки как "Абонент не существует", "Абонент не в сети", "Абонент заблокирован")


3. Q: What are the advantages of SMPP
   
   A: 
- This is a standard industrial protocol, no need to rewrite back-end if you move to another SMPP provider / ESME
- Asynchrony, high throughput
- Sending messages is almost instantaneous
- Prompt delivery of delivery status, submit errors are more informative, delivery time accuracy up to a minute
- The ability to check the number and location of the subscriber using HLR-request (eg there are such errors as "Subscriber does not exist", "Subscriber is offline", "Subscriber is blocked")


4. Q: How to get ESME status of sent message immediately / SMPP implementaion by fiorix.
   
   A:

sm, err = c.tx.Submit(m)
if err == nil {
    log.Println("Send > ID:", sm.RespID())
} else {
    log.Println("Send > err", c.err)
    if sm != nil {
        s:= uint32(sm.Resp().Header().Status)
        log.Println("Send > err #", s)
    }
}



5. Q: How to check if transmitter is bound ? / SMPP implementaion by fiorix
   I want to restrict Send() of message during reconnects. Because if Bind of Tranciever coincide with Submit_SM it causes Bind to fail -- SMSC does not answer on Bind.
   
   
   A: Solution:
   

   conn_st_ch := c.tx.Bind()
   go func() {
       for {
           select {
           case conn_st := <-conn_st_ch:
               c.bound = conn_st.Status() == smpp.Connected
               log.Println("bound>", c.bound, c.id)
           }
       }
   }()
   

   
   Note: It works because of the non-blocking nature of signal sent from client::notify
6. Q: What software I can use for Tests / Debug / Simulation of SMPP gateway ?
   
   A:
   I recommend ActiveXPerts SMPP simulator. However it doesn't work under wine.
7. Q: How to parse short message from deliver_sm_resp
   
   A: for JavaScript
   

   var re_key = /(\w+(\s\w+)?)/
   var re_val = /(\w+|[0-9a-f]+|null|\-)|(\s+|$)/
   re_key.compile(re_key);
   re_val.compile(re_val);
   
   // examples:
   //id:327 sub:001 dlvrd:001 submit date:1506272148 done date:1506272148 stat:DELIVRD err:null text:-'
   //id:c449ab9744f47b6af1879e49e75e4f40 sub:001 dlvrd:0 submit date:0610191018 done date:0610191018 stat:ACCEPTD err:0 text:This is an Acti
   //id:7220bb6bd0be98fa628de66590f80070 sub:001 dlvrd:1 submit date:0610190851 done date:0610190951 stat:DELIVRD err:0 text:This is an Acti
   //id:b756c4f97aa2e1e67377dffc5e2f7d9b sub:001 dlvrd:0 submit date:0610191211 done date:0610191211 stat:REJECTD err:1 text:This is an Acti
   //id:bd778cd76ae9e79da2ddc8188c68f8c1 sub:001 dlvrd:0 submit date:0610191533 done date:0610191539 stat:UNDELIV err:1 text:This is an Acti
   //
   //Field  Meaning
   //id  The message reference of the message.
   //    sub  Sub-ID, not used.
   //    dlvrd  Value '1' when the message has been delivered, if the message is still pending '0'.
   //    submit date  Submission date and time.
   //    done date  Date and time the status has changed, or message delivery time when stat is set to 'DELIVRD'.
   //    stat  Current status of the message.
   //    err  Additional error code, provider specific.
   //    text  Part of the original message text.
   
   module.exports.short_msg_parse = function(msg) {
       obj = {};
       var s = msg//.toLowerCase();
       for (;;) {
           var i = s.indexOf(':');
           if (i <= 0) break;
           var f = s.slice(0, i);
           if (k = f.match( re_key )) {
               var key = k[1];
               s = s.slice(i + 1);
               if (v = s.match( re_val )) {
                   i = v[0].length;
                   s = s.slice(i);
                   obj[key] = v[1];
               } else
                   break;
           } else
               break;
       }
       // TODO: validate field names
   
   FIELDS = [
           'id',
           'sub',
           'dlvrd',
           'submit date',
           'done date',
           'stat',
           'err',
           'text'
       ];
       return obj;
   }
   

   
   
   
   
   
8. Q: How to parse done date in short message from deliver_sm_resp
   
   A: for JavaScript
   

   // possible date formats:
   // yyMMddHHmm
   // yyMMddHHmmss
   // yyyyMMddHHmmss
   module.exports.smppDate_toLocalDate = function (str) {
       var _size = Math.ceil(str.length / 2),
           _ret  = new Array(_size),
           _offset;
   
       for (var _i=0; _i<_size; _i++) {
           _offset = _i * 2;
           _ret[_i] = Number(str.substring(_offset, _offset + 2));
       }
   
       _ret[0] += 2000;
       _ret[1] -= 1;
       return utils.applyToConstructor(Date, _ret);
   }
   

   
   
9. Q: What routing restrictions do countries have ?
   
   A:
   USA -- Please use E.164 format, as 14087525280 , additionally apart from the fact that the number appears to be a landline, for USA traffic you need to follow the guidance in this article. To summarise for USA traffic: * For P2P traffic you will need to use a Nexmo virtual number as sender. * For A2P traffic you will need to use either a Short Code or a Toll Free number/
   
   China
   
   France -- When sending to France with alpha sender IDs you must include within the message "STOP SMS AU 36184".

External links

1. https://github.com/fiorix/go-smpp