I'm working on a project in Python and there's a very obvious, almost textbook, spot to throw in some concurrency. I'm allergic to reading documentation and most of my code comes from modifying stackoverflow snippets so I perhaps naively googled "python threading" and implemented that. Imagine my surprise when I got literally no performance gain. So I looked into it and it turns out threads in Python are just fake threads, the entire program still runs in a single thread. You get all the overhead of concurrency with none of the benefits.
So I switched over to the multiprocessing library, which thankfully was pretty painless since it works almost identically to the threads library except with, you know, actual concurrency. That started out ok but I ran into trouble trying to return complex objects from the spawned processes. You can return simple data types pretty easily in a thread safe way, and even objects composed of simple data types, but trying to return an object that contains another object simply doesn't work. So once again I looked into it more and it turns out I have to specifically use a SyncManager if I want to return complex objects. So once again I implement that and it finally fricking works after hours of mucking around in the shit.
Why are there like 30 different ways to do concurrency in Python, with most of them sucking butt or not working at all? I've done concurrency in lots of other languages and they're all very straightforward. Even fricking C++ makes it easy to implement.
Jump in the discussion.
No email address required.
pp isn't a neet
Jump in the discussion.
No email address required.
Hi Maximus!
Jump in the discussion.
No email address required.
Jump in the discussion.
No email address required.
More options
Context
More options
Context
just an r-slur
Jump in the discussion.
No email address required.
More options
Context
More options
Context
Python really is an awful language to have as much sway as it does. The entire "let's prototype in Python and then rebuild it in a faster language" paradigm is extremely popular in a lot of the software industry, which goes to show how r-slurred it really is.
Performance "optimizations" in Python come from understanding how the interpreter/GIL/etc. works, which is a lot more complex than performance optimizations in real languages, where you just have to understand basic data structures and computing principles instead.
Jump in the discussion.
No email address required.
There are only two kinds of languages: the ones people complain about and the ones nobody uses.
Jump in the discussion.
No email address required.
More options
Context
Skill issue.
Vectorize yourself immediately.
Jump in the discussion.
No email address required.
More options
Context
More options
Context
Actually you are using multiple threads, the only thing is that there is a mutex lock on the interpreter meaning only one can run at a time
.
Jump in the discussion.
No email address required.
i bet he doesn't know why that's the case.
Jump in the discussion.
No email address required.
More options
Context
More options
Context
The GIL and its consequences
Jump in the discussion.
No email address required.
Can't you disable the GIL in the latest version?
Jump in the discussion.
No email address required.
There's lots of effort going on but I'm not sure that released yet. https://peps.python.org/pep-0703/
Looks like the next version should have it?
Jump in the discussion.
No email address required.
Oh, thought it was released already. Theres been talk of it for a few years now.
Jump in the discussion.
No email address required.
More options
Context
More options
Context
More options
Context
More options
Context
maybe you should have just used a simpler return format r-slur, this is a literal self-made problem
Jump in the discussion.
No email address required.
Jump in the discussion.
No email address required.
They really should
Jump in the discussion.
No email address required.
More options
Context
More options
Context
More options
Context
Jump in the discussion.
No email address required.
More options
Context
Meanwhile concurrency in go is based as heck. Go chads win again.
Jump in the discussion.
No email address required.
Jump in the discussion.
No email address required.
They are all
languages my man
Jump in the discussion.
No email address required.
More options
Context
More options
Context
More options
Context
Learn D
Jump in the discussion.
No email address required.
More options
Context
max_workers=1, 'cause I ain't in a rushJump in the discussion.
No email address required.
More options
Context
That sounds pretty r-slurred. How'd you get stuck with Python?
Jump in the discussion.
No email address required.
More options
Context
I'm trying to understand the absolute brain damage required to be using python and wanting performance
Jump in the discussion.
No email address required.
least energy efficient language on the planet!
Jump in the discussion.
No email address required.
More options
Context
processing list of things one thing at a time takes 10 seconds per thing
processing each thing in list in a separate thread takes 10 seconds total
#wow #whoa
Jump in the discussion.
No email address required.
You're counting at a granularity of seconds, yet you're using python. Certifiable skill issue
Jump in the discussion.
No email address required.
More options
Context
More options
Context
More options
Context
its a bit hacked on. python is ancient
Jump in the discussion.
No email address required.
More options
Context
Yeah the whole multiprocessing hack is one of the reasons I didn't wanna keep using Python
Jump in the discussion.
No email address required.
More options
Context
Jump in the discussion.
No email address required.
More options
Context
Python is heck with large projects (or anything beyond 1000 lines, really), and if the python code itself is so slow that you need multi-processing/multi-threading at all you'll be far better off just writing C++ or C, or literally any compiled language, because the baseline performance will be like 10x faster.
Lately I've been using Live++ for large C++ projects and it helps iteration times a lot, too.
Interpreted languages that rely purely on runtime type safety are garbage at scale.
Jump in the discussion.
No email address required.
More options
Context
Implement compute-intensive shit in other lang and make native python module or use FFI or something(lots of ways, just be creative tbh - python here is for high-level "management"). Python is cheapeasy lang and GIL was easiest(?) solution to shared resource access hazards. And well, concurrency isn't(?) exactly parallelism. And these "fake" threads can be helpful because you can do other things instead of waiting when there are some IO going on, etc - asynchronous shit. Idk much about python though.
Jump in the discussion.
No email address required.
More options
Context
Ok go use haskell or something if you want concurrency so bad
Jump in the discussion.
No email address required.
More options
Context
Can you explain why concurrency is necessary for your project? I'm sure you can find a janky way around it/""simulating"" it for future maintainers sake
Jump in the discussion.
No email address required.
More options
Context
Are you a literal NPC? First you had trouble counting to two, now you're just copy and pasting your replies. Do you need some time to update before you can type anything new?
Jump in the discussion.
No email address required.
More options
Context
Python has never EVER been about multi-threaded performance. It's why I strictly use it only as a script language or a very very small none computationally intensive app. Any bullshit about throwing cocurrence on top is just cope.
Jump in the discussion.
No email address required.
More options
Context
is that like bitcoin or something?
Jump in the discussion.
No email address required.
More options
Context
This. I just switched to C++ for my application.
Jump in the discussion.
No email address required.
More options
Context