Please login or register.
This location

Guys, consider to not jump into 0mq

aka-rider posted this 8 years ago

aka-rider | 8 years ago

Hello, I registered especially to try to make reconsider switching RPC to 0mq.

  1. 0mq was declined and abandoned by its creator
  2. After coredump of one of the peers all network may become unstable (data loss, peers stuck)
  3. Numerous memory leaks under heavy load
  4. Implementation does not allow high number of clients
  5. Does not work with proxy servers by design, and overall routing is bad (you have to have all peers with the "white" fixed IPs), if peer changes IP, network may become unstable
  6. REQ-REP: forces user to call read/write in the certain sequence. So if one peer was coredumped, counterpart is stuck and needs to be rebooted.
  7. This is not the right tool for the job.

The above statements were taken from the production environment.

The most of all: Using 0mq is a kind of vendor-lock. Now, having JSON-RPC I'm able to write app for Monero ecosystem in any language from C to Haskell.

Sorry, I couldn't find any discussion behind 0mq, I don't know what problem are you guys trying to solve. Please provide me some link, maybe I may be useful here.

Cheers, aka-rider

Replies: 11
fluffypony edited 8 years ago Replies: 2 | Weight: 0 | Link [ - ]

Ok, answers below:

  1. Pieter has cancer and is dying. He has very little time left in his life, and has had to hand the operating of the 0MQ project over to the community members that have been doing it for ages anyway. See: http://hintjens.com/blog:115
  2. I'm not sure how a backend system that the daemon doesn't even have to use (unless it has external programs interacting with it) would affect front-facing functionality at all. This point doesn't make much sense.
  3. I can't find any Github issues, mentioning memory leaks, that are unfixed or are more recent than 2014. Got a citation for that?
  4. Right now the JSON RPC maxes out at 4-5 instances of simplewallet on a reasonably well-specced machine. Anything above that starts affecting even automatic refreshes, and you've got no chance of quickly catching up on an unsynced wallet. 0MQ, on the other hand, maxes out at whatever your system's ulimit is, normally 1024 per program by default. If you have a specific use-case where you need more than 1024 wallets connected to the daemon, don't worry, you can just increase your system's ulimit.
  5. You keep using the word "peer", but I've got no idea what peers have to do with this. By definition this is for IPC, Inter-Process Communication. The fact that it uses local TCP instead of domain sockets is only to ensure cross-compatibility with Windows, which doesn't support domain sockets. This means that remote connectivity is possible, as we currently have with simplewallet's --daemon-address and related flags. But if your scenario requires complicated proxy routing to get to a remote daemon you should rethink your scenario, not try and force an IPC tool to work for complicated routing.
  6. That's not true; a REQ-REP can time-out if it's incomplete. Also if simplewallet crashes, for instance, then when you bring it back up it wouldn't be using the same connection, so it's irrelevant.
  7. What other IPC messaging systems are there that are as mature as 0MQ?

I'm not sure why you'd consider 0MQ a vendor lock. That's like saying HTTP is a vendor lock, and you want to be able to just send raw TCP/IP. 0MQ makes things significantly easier for future developers that write pool software or solo miners or wallets.

And you can use 0MQ bindings for everything from C: http://czmq.zeromq.org
to .NET: https://github.com/zeromq/clrzmq4
to Go: https://github.com/pebbe/zmq4
to Scala: https://github.com/zzxx-husky/zeromq-scala-binding
to C++: https://github.com/zeromq/cppzmq
to PHP: https://github.com/mkoppanen/php-zmq
to Fortran: https://github.com/scemama/f77_zmq
to Bash: https://github.com/marianov/zmq-bash-push
to Haskell: https://hackage.haskell.org/package/zeromq4-haskell

The problem we're solving is replacing the IPC layer with something designed to be used for IPC. There will always be a JSON RPC API, but it will be abstracted away into a separate binary, and will use 0MQ to communicate with the daemon under the hood.

Reply to: fluffypony
tewinget posted 8 years ago Replies: 2 | Weight: 0 | Link [ - ]

> You keep using the word "peer", but I've got no idea what peers have to do with this.

I think that may be referring to an assumption that the p2p code will be rewritten to use 0mq as well. I personally wouldn't be averse to that, but afaik that's not planned in the near-future. You'd know better than I would there though, fluffypony.

Reply to: tewinget fluffypony
fluffypony posted 8 years ago Weight: 0 | Link [ - ]

But that's replacing our wire protocol with ZMTP, not 0MQ (http://zmtp.org). Also that doesn't make sense because our wire protocol has never been JSON RPC...

Reply to: tewinget fluffypony
aka-rider edited 8 years ago Replies: 2 | Weight: 0 | Link [ - ]

> I'm not sure why you'd consider 0MQ a vendor lock. That's like saying HTTP is a vendor lock

Consider Python and Ruby. C library will lock GIL on a call, so it's not possible to handle more that one request at a time with threads, you have to use WSGI or multiple processes instead. It's not possible to use async event loops, like gevent. Same is true for the single threaded node.js - C library will block whole event loop. In golang it's even worse - you'll get whole OS thread locked on syscall, which means every goroutine in this thread will be locked. While native golang mutex reschedules goroutine, C library just blocks thread.

Next thing -- you have to rely on the 3rd party library implementation instead of using native TCP or HTTP library implementation. Quality of each library might be a big question.

I will answer other questions later.

Reply to: aka-rider tewinget fluffypony
aka-rider posted 8 years ago Weight: 0 | Link [ - ]

>I think that may be referring to an assumption that the p2p code will be rewritten to use 0mq as well.

Yes, it was the case

Reply to: aka-rider tewinget fluffypony
tewinget posted 8 years ago Weight: 0 | Link [ - ]

> you have to rely on the 3rd party library implementation instead of using native TCP or HTTP library implementation

Here's the thing: using any library we didn't write will be relying on a 3rd-party implementation of something. That's the definition of 3rd-party. I don't know anyone who says "don't use boost because it's 3rd-party to the C++ standard" -- though I do know people who say don't use boost for other reasons, but that's out of scope here.

Current code quality is independent of the author. Future code quality is always a concern regardless of author.

Reply to: fluffypony
aka-rider edited 8 years ago Replies: 2 | Weight: 0 | Link [ - ]

Thanks for the reply, fluffypony.

  1. https://www.infoq.com/news/2012/03/Crossroads-IO

  2. Ok. Here's why I was asking for some docs. I was afraid that you've decided to switch p2p protocol with 0mq.

  3. Our colleagues faced the leaks, but were not able to find a reproducible case. Just "time to time under heavy load". I was about a year ago. Maybe it is fixed now.

  4. >JSON RPC maxes out at 4-5 instances of simplewallet And what is the reason? I don't believe that the cause is in JSON parsing or protocol itself.

  5. By peer I mean 0mq socket.

  6. Oh. Timeouts were added later. I was not aware of that.

  7. This is the tough one. 0mq is mature, but it was never widely adopted. On the other hand, I can not name even one RPC library, while we constantly research this topic on my day job and have tried few of them.

So, my point is: you may try to use 0mq, but don't expect it'll magically cure all you problems. It's more like exchanging one set of known problems (like current performance) with another set of unknown.

Reply to: aka-rider fluffypony
tewinget edited 8 years ago Replies: 1 | Weight: 0 | Link [ - ]

So far I haven't had any issues with 0mq, but follow my branch for updates and give it a good stress test once there's enough functionality to test!

Reply to: tewinget aka-rider fluffypony
aka-rider posted 8 years ago Weight: 0 | Link [ - ]

I hope you'll succeed

Reply to: aka-rider fluffypony
fluffypony posted 8 years ago Replies: 1 | Weight: 0 | Link [ - ]

Some replies for you :)

  1. That article is a bit misleading. Crossroads I/O stalled on launch, and they ended up forking 0MQ into nanomsg, which ended up like this: http://sealedabstract.com/rants/nanomsg-postmortem-and-other-stories/ - meanwhile, 0MQ is still going strong and is better off without "rockstars" (https://hypatia.ca/2016/06/21/no-more-rock-stars/).
  2. We are looking at switching the p2p wire protocol to ZMTP (http://zmtp.org) instead of the weird, custom, Boost::ASIO protocol that it currently uses. Note that peers do not use JSON / HTTP on port 18080.
  3. Pretty sure it's fixed, haven't noticed anything in our testing.
  4. It's primarily due to the non-standard and poorly performant web server that is implemented. Replacing that with a high performance web server is a bit of a non-starter, we'd end up forking nginx;)

We seem to agree on the rest - shout if anything is unclear.

Reply to: fluffypony aka-rider fluffypony
aka-rider edited 8 years ago Weight: 0 | Link [ - ]

Thanks, fluffypony. Now I get a bit more understanding.

As for daemons IPC communication.

I use boost::asio with the text based (Memcached) protocol in my project -- cachelot.io Pretty simple implementation provides 20-25 kRPS from the single core but it doesn't scale well on OS threads. I will try the combination of C library and golang server on top of it. Golang performs very well when it comes to multithreading and networking.

I hope you guys will succeed with 0mq, but I sceptical about that.

>We are looking at switching the p2p wire protocol to ZMTP

I think, as a project, you have to grow Monero ecosystem instead of just solve technical problems. For now, Monero is a speculation asset, not a money. You need people willing to sell something for Monero; and if there is even small chance that ZMPT may avert adoption, you should abandon it.

I don't understand why to use ZMPT. ZMPT is a little known protocol, especially considering that adopting any binary protocol is hard, even HTTP/2 is on hold.

ZMPT allows to transfer binary data -- Google Protobuf was there for ages and it has implementation for every platform known (although, I wouldn't recommend protobuf either because generated stubs dictate code structure) ZMPT tries to be TLS -- but there is TLS, well tested, always under security audits, etc.

The following is a no-no for (almost) any financial company. Lawyer are mostly uncomfortable with the software licensing and want to be on a safe side.

ZMTP is thus licensed under the GPLv3, so that any derived versions must also be made available to users of software that implements it.

I would suggest to use some dumb JSON based protocol over TLS with compression instead; of course, I know little about the internals.