Open Call For Coders: Please Help Me Fix Autodrama!

https://fsdfsd.net/HeyMoon/autodrama/src/branch/master/autodrama.py

hey bot-fans, you may have noticed that fan-favorite bot @autodrama stopped posting about three months ago. This is because the monkeys at pushshift screwed up the way that comments are retrieved. I call into pushshift's API here to get comments on a post, however, this randomly stopped working.

PMAW is a python wrapper for pushshift. I only use it once, and that is to get all the comments on a particular post. The API documentation says that I can do this with "link_id" (https://pushshift.io/api-parameters/) but when I try to hit api.pushshift.io/reddit/comment/search/?link_id=10pv7qa&subreddit=ScienceUncensored, I get this response:

{
    "data": [],
    "error": null,
    "errors": {
        "error": {
            "root_cause": [
                {
                    "type": "query_shard_exception",
                    "reason": "failed to create query: For input string: \"10pv7qa\"",
                    "index_uuid": "htpvQm2RT4uCWzM53Q3zuw",
                    "index": "rc_2005-12"
                },
                <above object 200 more times>
            ],
            "type": "search_phase_execution_exception",
            "reason": "all shards failed",
            "phase": "query",
            "grouped": true,
            "failed_shards": [
                {
                    "shard": 0,
                    "index": "rc_2005-12",
                    "node": "r5briY5hS_mr4RzlALKdow",
                    "reason": {
                        "type": "query_shard_exception",
                        "reason": "failed to create query: For input string: \"10pv7qa\"",
                        "index_uuid": "htpvQm2RT4uCWzM53Q3zuw",
                        "index": "rc_2005-12",
                        "caused_by": {
                            "type": "number_format_exception",
                            "reason": "For input string: \"10pv7qa\""
                        }
                    }
                },
                <above object 200 more times>
            ]
        },
        "status": 400
    }
}

Yes, I reported this issue to the maintainers of pushshift, but have gotten no response back.

Clearly, the backend is interpretting what should be strings as numbers, because this works: api.pushshift.io/reddit/comment/search/?link_id=100000&subreddit=trees

I also tried using pushshift's /comment_ids endpoint but shit's broken too.

Fellas, any thoughts? @automeme is my only child that is not estranged, unlike my other children @bbbb (who hates me 😭) and @automeme (who is clinically retarded)

Jump in the discussion.

No email address required.

MayoAntiRacist I/me 1yr ago #3600302 Edited 1yr ago

It says number format exception, so it seems that it expects link_id to be an integer rather than a string of letters and digits.

The API documentation also says that link_id is an Integer. Apparently Reddit uses base 36 IDs, so you'll probably need to convert 10pv7qa from base 36 to base 10 and use that as the link_id argument.

Yep. This works:

https://api.pushshift.io/reddit/comment/search?link_id=2220229090&subreddit=ScienceUncensored

Edit: Actually, no, it doesn't work. That succeeds for 12 shards but fails for the rest with errors like this:

{"total":828,"successful":12,"skipped":0,"failed":816,"failures":[{"shard":0,"index":"rc_2005-12","node":"sYpN84pmQYuHQmYijuzUeg","reason":{"type":"query_shard_exception","reason":"failed to create query: Value [[32 32 32 30 32 32 39 30 39 30]] is out of range for an integer","index_uuid":"htpvQm2RT4uCWzM53Q3zuw","index":"rc_2005-12","caused_by":{"type":"illegal_argument_exception","reason":"Value [[32 32 32 30 32 32 39 30 39 30]] is out of range for an integer"}}},

I guess it's complaining because 2,220,229,090 is out of range for a 32-bit signed integer. But maybe that's okay, because it's only failing for older shards that predate the post anyway. You might want to limit your query so that it doesn't search for comments older than the post itself.

Edit: Got it working-ish here

14 Context

HeyMoon hey/moon :marseyfoxgloveyourself:

touch foxglove NOW :!marseyfoxgloveyourself:

MayoAntiRacist 1yr ago #3600402

well that's the other thing, if you look in the bottom of the response you will see other errors...

{
            "shard": 0,
            "index": "rc_2009-07",
            "node": "r5briY5hS_mr4RzlALKdow",
            "reason": {
              "type": "query_shard_exception",
              "reason": "failed to create query: Value [[32 32 32 30 32 32 39 30 39 30]] is out of range for an integer",
              "index_uuid": "IOqjwfolQCmRF2dyKMhkUA",
              "index": "rc_2009-07",
              "caused_by": {
                "type": "illegal_argument_exception",
                "reason": "Value [[32 32 32 30 32 32 39 30 39 30]] is out of range for an integer"
              }
            }
          },

I think some of the shards are failing due to the index being out of range so you only get an incomplete set of comments. So, from this command we only get 10 comment when there are 104 comments in the original post...

4 Context

MayoAntiRacist 1yr ago #3600306

that's almost certainly what's going on but idk what to do to work around it :marseydepressed:

3 Context

MayoAntiRacist I/me HeyMoon 1yr ago #3600537

This gets all the comments for that post:

https://api.pushshift.io/reddit/comment/search?link_id=2220229090&after=1673802423&size=150&subreddit=ScienceUncensored

The reason you only got ten comments was that that's the default page size. The errors were unrelated; you weren't going to get any comments from those shards because they predate the post anyway. So when you query, just set the "after" parameter to the time the OP was posted, and it won't search the older shards, so you don't get any errors. Then by setting the "size" parameter, you can increase the page size. I believe it's to a maximum of 500. There's presumably a way to page through the comments, but you'll have to check the docs for that.

7 Context

MayoAntiRacist 1yr ago #3600810

oh shit, based!

DukeHerbertOfFucksticks deb/ian platydown

drunk as a lord HeyMoon 1yr ago #3600386 Edited 1yr ago

Can't you just use something like https://pypi.org/project/base36/ and use it when constructing query params?

In here? https://fsdfsd.net/HeyMoon/autodrama/src/branch/master/autodrama.py#L115

Go here, int() can do it without a library: https://stupidpol.site/h/programming/post/147577/open-call-for-coders-please-help/3600378#context

DukeHerbertOfFucksticks 1yr ago #3600410

ye you can do int(x, 36) but there is another issue (commenter editted his comment, see my response to this suggestion here

2 Context

drunk as a lord HeyMoon 1yr ago #3600419

If shards are fricked then I don't think you can fix it without upstream's intervention. :marseysad:

DukeHerbertOfFucksticks 1yr ago #3600431

despair

:marseydespair:

goodNews4U she/her I love all God's children! 🥰🥰🥰 1yr ago #3600437

Hey, you idiot. It's not our fault that your bot stopped posting. It's because you're using a shitty Python wrapper that doesn't work. So go screw yourself, and learn how to use the API properly.

11 Context

goodNews4U 1yr ago #3600448

cute twink, it's not the wrapper's fault that ht escript stopped working, it is the API's fault. Frick you queer

goodNews4U she/her I love all God's children! 🥰🥰🥰 HeyMoon 1yr ago #3600501

That's a really ignorant thing to say. Maybe if you did some actual research you would know that it's not the wrapper's fault, but the API's. But I guess that would require too much effort for you.

SnappyIsMyWaifu gonna/say/it 🇮🇱\ :marseykneel:

/🇷🇺 1yr ago #3600285

Lmao, I barely even do the coding I'm paid to do

SnappyIsMyWaifu 1yr ago #3600301

I am paid to do coding but they tell me not to which is why I do it anyways :marseymad:

omg muh hecking regressions

muh scrum

muh story points

AAAAAAAAAAAAAAAAAA FRICK AGILE I JUST WANNA COOOOOOOOOOOOOOOOOOOOODE

icebornCamel reeee/tard Brussel sprouts are an abomination to the good name of vegetables HeyMoon 1yr ago #3600626

#relatable

Mystic_Gohan E/Celeb SnappyIsMyWaifu 1yr ago #3600382

yeah but this is rdrama.net bro it's actually important

5 Context

1yr ago #3600291

jc aevann sneks !!

mineral mar/sey :marseyravegigaspeed:

1yr ago #3600378

https://old.reddit.com/r/pushshift/comments/103k1qe/anyone_have_luck_using_the_link_id_param_in_the/j2zyjkp/

To query by link_id using the new API, you must (at least right now) convert from base 36 to base 10. So for this submission, 103k1qe converted to base 10 is:

https://api.pushshift.io/reddit/search/comment?link_id=2182756550

Python:

int('103k1qe', 36)

JavaScript:

parseInt('103k1qe', 36)

Recent submissions will generate errors from older shards, it appears that you can ignore them. They apparently occur (this is new to me) on shards containing comments from older submissions (before Dec. 2023 / before link_id 231).

mineral 1yr ago #3600427

unfortunately, same result for newer posts as I described here, with incomplete results. but, I am glad other people know about this!

HeyMoon 1yr ago #3600450

Yeah, the comment addresses this at the end. I don't think there's anything you can do about it. The maintainer of the API seems to be in the middle of reindexing his ES cluster.

mineral 1yr ago #3600465

>tfw automeme could be back by monday and i literally just have to sit on my butt and wait for it to happen

codechads, its a good day

mineral 1yr ago #3600413

interesting!!!! let me give this a try

drunk as a lord mineral 1yr ago #3600407

Since when Python had base as a second parameter for int()? TIL.

C333 Dino/Dinos :marseycapydinosaur:

1yr ago #3600420

Fellas, any thoughts? @automeme is my only child that is not estranged, unlike my other children @bbbb (who hates me 😭) and @automeme (who is clinically r-slurred)

:#marseythonk:

goodNews4U she/her I love all God's children! 🥰🥰🥰 C333 1yr ago #3600432

:marseyeyeroll:

ForcedRegistrycel Man/let Proud white father of a Black child 1yr ago #3600294

Have you tried turning it off and on again?

ForcedRegistrycel 1yr ago #3600304

yes.

6 Context

TheOverSeether cute/twink :marseycrusader:

1yr ago #3600535

:#marseyquestion:

Did you try adding Marseys to it?

pm-me-manifestos eval/apply I promise I'm not like this IRL 1yr ago #3603448 Edited 1yr ago

I started reading through your code and this is the first non-trivial function I see.

def get_comment_basedness_out_of_five(basedness: int, absolute : bool):
    if (absolute):
        if basedness > 1000:
            score = 5
        elif basedness > 500:
            score = 4
        elif basedness > 100:
            score = 3
        elif basedness > 50:
            score = 2
        elif basedness > 10:
            score = 1
        else:
            score = 0
    else:
        if basedness > 100:
            score = 5
        elif basedness > 50:
            score = 4
        elif basedness > 10:
            score = 3
        elif basedness > 5:
            score = 2
        elif basedness > 1:
            score = 1
        else:
            score = 0
    return get_score_string(score, "🔥", "🔘")

:marseyretard3:

pm-me-manifestos 1yr ago #3603607

lol, hey if it works it works. easier that defining a new function every time I want to change the algorithim

cumlord piss/shit AOC's Gynecologist 1yr ago #3602378

It looks like the API is returning an error message indicating that it failed to create a query because of an input string issue. Specifically, the error message says "For input string: "10pv7qa"", which suggests that the value you're passing for the "link_id" parameter is not in the correct format.

According to the Pushshift API documentation, the "link_id" parameter should be in the format of a base-36 Reddit ID. This means that you should remove the "t3_" prefix from the ID and pass the remaining string as the value for "link_id".

For example, if the ID of the post you're trying to retrieve comments for is "t3_10pv7qa", you should pass "10pv7qa" as the value for "link_id".

Try modifying your API call to use the correct format for the "link_id" parameter and see if that resolves the issue.