Threaded Web COMP

Here we have a component completely based of Elburz’s Twitter Component (Thank you very much for the inspiration) which was posted here: https://forum.derivative.ca/t/threaded-twitter-stream-search-tox/7165/1

In creating this component I tried catching as many cases as I can for communicating with different resources on the web.

So this let’s you control the Request Type (choose from GET, POST, PUT, DELETE, HEAD and OPTIONS), Stream the incoming data, control the Chunk Size for the Stream.
You also have the option to Authenticate the connection with Basic, Digest, oAuth1 or oAuth2 Authentication. The received data is being passed to a “callback DAT” in which you can deal with it any way you like.

Inputs to the COMP are a header Table DAT which should contain rows with Header name and value pairs. The second input is for the data you would like to send. Especially when sending a request or trying to post data you can also utilize the

op('webComp').Send(data={'myFile':b'somedata'})

function. This makes it possible to post anything from form fields to full images to a web resource.

When using oAuth1 for example to get the Stream from Twitter, you will have to obtain an App Key, App Secret and an oAuth Token as well as a oAuth Secret which ca be obtained from apps.twitter.com/ after creating and registering a new application.

Similar if you like to use oAuth2 with for example Facebook, you will have to supply an Client ID, Client Secret and a Token. To obtain the Token you will have to implement your own authentication steps or obtain a Client Token from the Apps Advanced Setting page at developers.facebook.com

I hope this can be a useful tool for the interim. It is probably missing features and I would love to hear about them or see your implementation!

Cheers
Markus
webCOMP.zip (4.2 MB)

Thanks for this.
I know it’s difficult to give working examples without sharing auth credentials but it would be great if it came with a quick rundown of settings or examples for other common operations on common platforms. For example getting instagram pics, facebook users, twitter locations etc. There is of course an undelying amount of API programming left to the user but any examples would be appreciated.

Wow :open_mouth: So clean and tidy. I’d rather use yours then mine :laughing:

Small update posted above. After long head scratching I learned that threads should not spin… So in this version I added a threading event which keeps the thread from eating up CPU cycles until a request is send from the outside.

Cheers
Markus

Accessing Twitter

  • go to apps.twitter.com and click the Create New App Button
    Step1.png
  • fill out the application details, the website can be a dummy address
    Step2.png
  • go to “Keys and Access Tokens” and click “Create my access token”
    Step3.png
  • get App Key, App Secret, User OAuth Token and User OAuth Secret from Consumer Key, Consumer Secret, Access Token and Access Token Secret
    Step4.png
  • set Authentication Type of the web Component to OAuth1 and fill in the parameters

Step5.png
connecting to Streaming API of Twitter:
For an overview of the available parameters refer to Twitter Documentation here

  • on the Web Parameter Page of the COMP set Request type to “Post”, the URL to “https://stream.twitter.com/1.1/statuses/filter.json” and turn on the “Stream” parameter.
    Step6.png
  • add a Table DAT to the second Input of the COMP and specify any of the request parameters listed in the twitter api documentation
    Step7.png
  • toggle the “Activate” parameter and pulse the “Go” parameter. In the textport you should see messages coming in.
    Step8.png
  • the incoming data is send to the docked DAT called web1_callback. Here it literally just prints our a dictionary with the keys
    [list]
    [*]msg
  • status
  • code
  • encoding
    [/:m]
    [
    ]while the stream is active there will only be the msg key and it’s value containing whatever comes down the stream[/:m]
    [
    ]to decode the twitter message enter this in the web1_callback DAT:
import json
def receive(data):
	msg = data.get('msg',None)
	if msg:
		decoded = json.loads(msg.decode())
		for k in decoded.keys():
			print('key:',k,'value:',decoded[k])
	return

Step9.png[/:m]
[
]to stop the stream, pulse the “Stop” parameter.[/:m]
[
]changing the request parameters only takes effect after stopping and restarting the stream.[/*:m][/list:u]

connecting to REST API of Twitter:
For an overview of the available connection points refer to Twitter Documentation here

  • To see the last tweets of a twitter account, choose the connection point GET statuses/user_timeline by entering the url api.twitter.com/1.1/statuses/user_timeline.json into the “URL” parameter of the COMP.
  • Choose the GET type in the “Request Type” parameter.
  • Turn of the “Stream” parameter
  • add a Table DAT to the second Input of the COMP and specify any of the request parameters listed in the twitter api documentation (for example “screen_name”)
  • Pulse the “Go” parameter
  • To decode this response enter this code into the web1_callbacks DAT:

[code]import json
def receive(data):
msg = data.get(‘msg’,None)
if msg:

	msg = msg.decode()
	decoded = json.loads(msg)
	for i in decoded:
		print('-------------------------')
		for k in i.keys():
			print('key:',k,'value:',i[k])
	return[/code]

Accessing Facebooks Graph API

  • go to https://developers.facebook.com/ and click the “Add a new App” from the “My Apps” dropdown.
    step1.png
  • fill out the application details and click “Create App ID”
    step2.png
  • go to “Settings>Basic” and note the “App ID” as well as “App Secret”
    step3.png
  • click on “Tools & Support” and then click on the “Graph API Explorer”
  • choose your app from the Application drop down
    step4.png
  • copy the “Access Token”
    step5.png
  • on the Web COMP select OAuth2 on the “Authentication Type” parameter and fill in the previously collected App Id, App Secret and Token
    step6.png
  • refer to the Graph API on how to retrieve information from the Facebook Graph

get your own feed

  • set the Web COMP “Request Type” parameter to “Get”
  • enter graph.facebook.com/v2.7/me/feed into the “URL” parameter
  • pulse the “Go” parameter
  • replacing “me” with a user id will return that users feed

Just wonderful Markus, can’t thank you enough. This is really making TD much more valuable for commercial projects that need custom social media displays.

It seems to have trouble with the json decoded callbacks provided. Won’t start thread, intermittent.

Can you give me instructions on how to duplicate?

When I enter your code into the callbacks dat :
import json
def receive(data):
msg = data.get(‘msg’,None)
if msg:
decoded = json.loads(msg.decode())
for k in decoded.keys():
print(‘key:’,k,‘value:’,decoded[k])
return

I get very intermittent functionality. I’ll try to dig up a toe when I get the time

I am getting this error, and don’t really understand why…
any idea?

{‘status’: ‘Thread Started’}
python >>>
{‘status’: ‘##start post request’}
python >>>
{‘code’: 401, ‘encoding’: ‘ISO-8859-1’}
python >>>
{‘msg’: b’\n\n\nError 401 Unauthorized’}
python >>>
{‘msg’: b’’}
python >>>
{‘msg’: b’’}
python >>>
{‘msg’: b’

HTTP ERROR: 401

’}
python >>>
{‘msg’: b"

Problem accessing ‘/1.1/statuses/filter.json’. Reason:"}
python >>>
{‘msg’: b’

    Unauthorized
’}
python >>>
{‘msg’: b’’}
python >>>
{‘msg’: b’’}
python >>>
{‘status’: ‘##end of message’}
python >>>

Hi gionniz.

what is it that you are trying to do?
You get a default “Unauthorized” response from the server indicating that your credentials are not correct//

Cheers
Markus

The question came up how to post images to Twitter.
I changed the Send method a bit to allow for specifying the url for each Send you are doing - this is especially useful when having to respond to events received from a server. Please download the latest WebComp zip from the first post.
First you will need to read in the file, then upload the file and finally post a status update with the id which twitter returns after successfully having received your image.

Here is a little guide:
1. Follow the Steps here: to create your Twitter Application and get all necessary OAuth info.

2.The Imagedata needs to be base64 encoded before it can be send to Twitter:

import base64

# save out a TOP
file = op('noise1').save('noise1.jpg')

# open the saved out file in binary format and read it into a variable
imgFile = open(file, 'r+b')
imgData = imgFile.read()

# encode as base64 and convert into a string
b64Data = base64.b64encode(imgData)
dataString = b64Data.decode()

# create a data dictionary you want to send to twitter
dataDict = {}
dataDict['media_data'] = dataString
dataDict['Name'] = 'An Image from TouchDesigner'

# send the data to Twitter
op('web1').Send(data=dataDict, url='https://upload.twitter.com/1.1/media/upload.json')

3. Now we wait for an response from Twitter and post an status update with the img id.
For this we need to create a script in the web1_callbacks DAT:

import json
def receive(data):
	# see if the data contains a message part
	msg = data.get('msg',None)
	if msg:
		# the returned message will be binary so lets decode this and convert the json into a dict
		msg = msg.decode()
		answerDict = json.loads(msg)

		# see if we can retrieve a media_id
		image = answerDict.get('media_id',None)
		if image:
			# create a data dictionary which contains the status update text and the uploaded media id
			dataDict = {}
			dataDict['media_ids'] = image
			dataDict['status'] = 'Testing Image Upload from TouchDesigner099'
			# send it to twitter
			op('web1').Send(data=dataDict, url='https://api.twitter.com/1.1/statuses/update.json')
		else:
			print(answerDict)
	else:
		print(data)
	return

Be advised that there is a limit on how much you can upload. So don;t run these scripts every frame. Also keep your Twitter applications OAuth keys and secret save!!!

Sorry for the newb question, anyone able give me a pointer on how to get the twitter stream python data that is successfully going into textport to a text DAT?

Something I can use to then select just the message and eventually send it to a text TOP?

Bit stuck on this one.

Thanks in advance

The supplied callbacks are creating a python dictionary of all keys/values and printing them to the console. You simply need to find which keys you would like to use and assign their value to whatever operators you would like. I would suggest reading through how to parse python dictionaries/lists.

Below is a simple callback eample that assigns the user’s screen name and tweet to Text TOPs…

import json def receive(data): msg = data.get('msg',None) if msg: decoded = json.loads(msg.decode()) for k in decoded.keys(): if k == 'user': print(decoded[k]['screen_name']) op('text1').par.text = '@' + decoded[k]['screen_name'] if k == 'text': print(decoded[k]) op('text2').par.text = decoded[k] print('-------------------------') return

Hi everyone! Thanks for putting together this COMP - this has helped open up a world of possibilities for connecting projects to web sources.

I’m working on an interactive projection activation, and we’re trying to build a very simple trigger to start playback on movie files, activated by a Twitter feed. We have a third party app that will write the tweets on a @“testAccount” we’ve set up with the appropriate developer API’s, we have our app key/secret/OAuth token/secret in place in the COMP, and we’ve entered the Twitter account in table4 (where @nasa appears in the original). We even beat some initial 401 Unauthorized issues. It seems like everything should be working, but…

After entering our @“testAccount” in table4, switching web1 to Active, and pulsing Go, we get our code 200 back, but then the script seems to hang with no feedback. Pulsing Stop or switching Active to Off (on web1) does not break the loop, and requires a full restart of TD to regain control of Textport.

python >>> {'status': 'Thread Started'} python >>> {'status': '##start post request'} python >>> {'code': 200, 'encoding': None} python >>>

Followed by an indefinite hold (10min+ tested)

This only seems to specifically be a problem with our “fresh” Twitter account we created specifically for this activation. Testing @nasa yields no problems - the following Textport comes from only changing our tracked Twitter handle in table4 from @“testAccount” to @nasa (stop initiated by pulsing Stop in web1).

python >>> {'status': 'Thread Started'} python >>> {'status': '##start post request'} python >>> {'code': 200, 'encoding': None} python >>> {'msg': b'{"created_at":"Tue Feb 21 18:58:41 +0000 2017","id":834115142975705095,"id_str":"834115142975705095","text":"Nasa announces major press conference on a \'discovery beyond our solar system\' https:\\/\\/t.co\\/HWG2qPOdXy","source":"\\u003ca href=\\"http:\\/\\/twitter.com\\" rel=\\"nofollow\\"\\u003eTwitter Web Client\\u003c\\/a\\u003e","truncated":false,"in_reply_to_status_id":null,"in_reply_to_status_id_str":null,"in_reply_to_user_id":null,"in_reply_to_user_id_str":null,"in_reply_to_screen_name":null,"user":{"id":3012613095,"id_str":"3012613095","name":"Barfield Educational","screen_name":"BarfieldEd","location":"Batley and surrounding areas","url":"http:\\/\\/www.barfieldeducational.co.uk\\/","description":"Providing STEM sessions in schools and social enterprise for disadvantaged children and senior citizens","protected":false,"verified":false,"followers_count":51,"friends_count":147,"listed_count":16,"favourites_count":25,"statuses_count":1243,"created_at":"Mon Feb 02 17:51:27 +0000 2015","utc_o python >>> {'status': 'Execution Interrupted, waiting for new commnad'} python >>> {'status': '##end of message'} python >>>

Some other accounts seem to work just fine - we’ve tried with @cubs and @Yankees (accounts selected arbitrarily), which both worked with no issues (maybe an extra second or two of latency on @Yankees?).

However, we had a similar-but-different issue when we tried to access @AppleSupport - there is roughly a ~20 second latency between Activating/pulsing Go web1 and receiving tweets. Similarly with sending pulses/Offs, Textport holds for 15-20 seconds before returning the Execution Interrupted. Maybe there’s a connection?

We’ve recreated this issue identically on both a laptop running Windows 10 and a desktop running Windows 7 SP1. We’re using 099 Commercial, but also have access to 088.

So. With all this in mind - what could we be possibly doing wrong here? Is our testTwitter account not configured properly? Yet that’s where we got our OAuth tokens etc from. Any suggestions or insight?

Thanks in advance!

I just tried getting tweets from one single account and made following changes:

The Request Type parameter was set to “GET”
The url to fetch my accounts tweets was set to “https://userstream.twitter.com/1.1/user.json

I disconnected the table4 from the input as I don’t want to filter for certain messages.

And it works well for me - but this only let’s me see my own accounts tweets or tweets mentioning my account.

For the behavior you describe, did you change the URL?

Cheers
Markus

HELP
Screen Shot 2017-04-02 at 4.33.05 PM.png

requests seems to not be running on OSX? Will try at the office on a mac…

Cheers
Markus

I’m getting similar errors trying to run this on a Mac so would be great to hear of a solution.