How to use Trade Me’s API with Python

Update: TradeMe has revoked my API access and is refusing to provide me with new credentials. Using it to make these blog posts was denied. As such I cannot check if this code still works. My apologies if it no longer works.

TradeMe has an excellent API that allows users to pull all sorts of data from their site, such as car or real estate listings. As a Python beginner, I was a bit intimidated by Python’s API and the authentication process. However, it turned out to be very easy to use with the help of Python’s “OAuth1Session” library. 

Step 1: Register your program with TradeMe

All you need to do to get started is sign into TradeMe and fill out this form on their website: https://www.trademe.co.nz/MyTradeMe/Api/RegisterNewApplication.aspx
TradeMe got back to me within a couple days and approved my application. Once approved, you can see your consumer Key and Secret at: https://www.trademe.co.nz/MyTradeMe/Api/DeveloperOptions.aspx

Step 2: Authenticate yourself 

In order to request information from TradeMe, you need to authenticate yourself so TradeMe knows who is asking for data. Although it may be tedious and has to be done manually, I found it easiest to authenticate myself using their web-page: https://developer.trademe.co.nz/api-overview/authentication/

Simply put in your consumer key and token, select “Production” for the environment (so you get data from the real TradeMe, rather than their fake sandbox site for testing purposes), and select the permissions you need. Generate your secret Key and Token and copy them into a Python script. 

Keep your consumer and secret Key, Token, and Secrets private! Otherwise, anyone could impersonate you on TradeMe. 

Step 3: Using Python to make requests

In order to request data from TradeMe, we need to make authenticated searches to their website. I am using the Python library “requests_oauthlib” to handle all the heavy lifting. TradeMe uses OAuth 1.0 to handle authentication. Since TradeMe uses OAuth1.0, we can use “OAuth1Session” from “requests_oauthlib”. We pass in our consumer key, token, and secrets into “OAuth1Session” and start making some requests! 

Example search

I’m interested in buying a new car but don’t want to pay too much. I could manually comb through all of TradeMe’s car listings and get a general idea of the price of cars I’m interested in are. OR I could have Python do it for me and get the exact values of specific makes and models. Here’s example code which uses OAuth1Session to ask for all the current car listings on TradeMe in JSON format then stores them using Pandas. You can make your own TradeMe searches here: https://developer.trademe.co.nz/api-reference/search-methods/

#Import needed libraries
from requests_oauthlib import OAuth1Session
import time
import json
import pandas as pd

# Keys (found on your TradeMe account page)
consumerKey = 'yourConsumerKeyHereAsAString'
consumerSecret = 'yourConsumerSecretHereAsAString'

# Easy way to get these here: https://developer.trademe.co.nz/api-overview/authentication/
oAuthToken = 'yourTokenHereAsAString'
oAuthSecret = 'yourSecretHereAsAString'

# Use oAuth to authenticate ourselves with our keys and tokens
tradeMe = OAuth1Session(consumerKey,consumerSecret,oAuthToken,oAuthSecret)

# Tradme uses HTTP for get requests, meaning you ask for specific data by putting in a long URL
searchCars = 'https://api.trademe.co.nz/v1/Search/Motors/Used.json?body_style=NotSpecified&condition=All&listed_as=All&listing_type=All&page=1&photo_size=Thumbnail&rows=500&shipping_method=All&sort_order=Default&transmission=NotSpecified HTTP/1.1'

# Get the search page
returnedPageAll = tradeMe.get(searchCars)

# Parse the data
dataRawAll = returnedPageAll.content
parsedDataAll = json.loads(dataRawAll)

# Total number of listings
totalCount = parsedDataAll['TotalCount']
# TradeMe limits to 500 listings per request 
totalRequests = int(totalCount/500)

# For all the requests it takes to get all the data
for i in range(0,totalRequests+1):
    pageNum = str(i)
    # Set up search for each page
    searchAll ='https://api.trademe.co.nz/v1/Search/Motors/Used.json?body_style=NotSpecified&condition=All&listed_as=All&listing_type=All&page=' + pageNum + '&photo_size=Thumbnail&rows=500&shipping_method=All&sort_order=Default&transmission=NotSpecified HTTP/1.1'
    returnedPageAll = tradeMe.get(searchCars)
    dataRawAll = returnedPageAll.content
    parsedDataAll = json.loads(dataRawAll)

    # Get the Listings
    eachListingAll = parsedDataAll['List']

# Convert to Pandas
    pandaAll = pd.DataFrame.from_dict(eachListingAll)

# Save data
    if i == 0: # If first search
        pandaAll.to_pickle('dataCars.pkl')
    else: # Append existing data
# Load from storage
        pandaAllStorage = pd.read_pickle('dataCars.pkl')
    
# Append storaged with new data
        pandaAllStorage = pandaAllStorage.append(pandaAll,ignore_index=True)

# Save (will overwrite old stored data)
    pandaAllStorage.to_pickle('dataCars.pkl')
    time.sleep(0.5)
    print(i)

print('done')