[Twisted-Python] verilog simulation calling out to twisted

[ex vito]

On 2017-09-20, at 22:33, Bryan Murdock <bmurdock at gmail.com> wrote:

> I've been writing Verilog simulations for years, and tinkering with
> Python for years.  In the last couple years I have been calling out to
> Python from Verilog simulations, which has been awesome.  Just this
> week I tried twisted out for the first time to write a simple custom
> TCP server.  It was a great experience and even though I still know
> next to nothing about twisted, I think I'm in love :-)

Welcome to the club! :)


> Now I have a crazy idea.  I'd like that Python code that my Verilog
> calls out to to use twisted.  OK, actually I've already done that, my
> Verilog calls out to my that TCP server that I wrote.  I had to spawn
> a thread and run the server in that thread so that the Verilog could
> continue to do stuff in parallel with the server.  It's working great.

Can you clarify what "your Verilog" is? Python code running a Verilog simulation, maybe? Something else?


> The real crazy thing I want is for the Verilog to call out to a
> twisted TCP client as well.  Again, I need the client and server to
> not block the Verilog.  My first attempt was to run client and server
> in two separate threads.  I start the server thread, then I start the
> client thread and only in the client thread do I call reactor.run().
> It seems to not be working.  Before I try to figure out why, I thought
> I might ask here if it even should work.

I do not fully understand this: if "Verilog calls out to TCP server" means it establishes a TCP connection (does it?) what does "Verilog calls out to a TCP client" mean? Does it call a function/method with async Twisted code? How does it handle the fact that the result is async?

In general you are able to run concurrent (not parallel) code for TCP clients and servers within the same thread - after all, that's the key idea behind async programming with Twisted, with the reactor driving concurrent handling of multiple connections and events.

Of course, again, in general, you'll want your code not to block and to do its things as fast as possible so that control can be returned to the reactor (so that it is ready to quickly handle other events).

A few notes about threads and Twisted:
- IIRC the reactor should (must?) be running on the main thread.
- Threads are a good fit to call out to "I/O blocking code" from Twisted.
  (talking to a database or doing heavy disk I/O, for example)

A few notes about threads and Python:
- Python (CPython that is) does not run two CPU-bound threads in parallel.
  (IOW, it can't simultaneously use two CPUs/CPU-cores in the system)
- That's due to the GIL: there's lot's of information about it on the internet.
- You will get lower I/O responsiveness even with a single CPU-bound + one I/O bound thread.


If "Verilog" needs to continue working and is, I assume, CPU-bound, you will probably be better off by having that run in a process and your Twisted code in a single-threaded separate process.

Benefits:
- Better separation of concerns.
- No mixing of sync/async code which at times can lead to complex design/code.
- No shared memory between "Verilog" and Twisted code.
- "Verilog" can use as much CPU as it needs.
- "Twisted" will be as responsive as possible.
- Easier to test.

Costs:
- Need to define how to start both processes.
- Need to define how those processes communicate.
- All communication between processes needs to be serialized (no shared memory).


These are a few general ideas you may want to consider. Like I said, I couldn't fully grasp where you're currently at and what exactly you're trying to achieve but hopefully these comments will serve as guidance.

Cheers,
--
exvito