About a year ago one of my friends introduced me to the online trivia LearnedLeague (LL for short). You have to be invited by a current member but then you are assigned a “rundle” or group of people about your level to play against and answer trivia, etc, etc. We not have 8 different friends playing and have a groupme chat going to discuss answers each day (after submitting of course). But it got kinda annoying to log in to LL each morning and navigate to the four different rundles we’re in to see the results or compare us amongst ourselves. So I used my web scraping and python to do it for me!
Basically, I wanted a program that I could run on the extra laptop in the office that at 6am everyday would run, fetch the stats for each of us from the website, make an image, then upload the image to our groupme account so that we could all see it when we woke up. I started trying to use BeautifulSoup for the scraping but the LL website is a mess with tables in tables in tables and to make matters worse, you have to log in to see the stats. I tried to use urllib to log in and ran into problems. So I switched the entire process to Selenium. Selenium made it super easy to login and get the data from the website and had lots of examples to make things easy. 10/10: would recommend.
If you chose to do something like this yourself, please take it easy on the LL website. My program checks once per day and only checks 8 people so I don’t think it’s much of a load on the site. But if somebody was checking thousands every minute multiple times a day, it would really mess things up. Don’t be that guy.
So lets break the program down into bite sized pieces and I’ll address each of those before putting the final/complete code at the bottom.
- Setup the program
- Log in to LL and scrape the data
- Sort the players based on rundle (A-E) and rank within the rundle
- Create an image of the sorted data
- Upload the image to the groupme chat
Setup the program
Here, we setup the program with the imports and then create a class called Player that stores the name and stats. The class also has a function to draw the data on an image to reduce copy/pasting later in the program. Then, in the actual program, we start by creating a list of Players and then adding the people I know to that list. This could have been done in one line but I wanted it to be easier to read, especially if I add more people in the future.
# LearnedLeage Automated Poster
# Created by Matt Walker on July 7, 2018
from selenium import bdriver
from selenium.webdriver.common.keys import Keys
from PIL import Image, ImageDraw, ImageFont
import os
import subprocess
import pyautogui
import pyperclip
import time
# Player class to store stats and draw them in an image
class Player:
name = ""
rundle = ""
season_rank = ""
season_points = ""
season_wins = ""
season_losses = ""
season_ties = ""
season_MPD = ""
season_TCA = ""
season_def = ""
last = ""
def __init__(self, label):
self.name = label
# Function to make drawing stats easier, takes and x and y location and image
def printStats(self, x, y, img_draw):
img_draw.text((x, y), self.name, font = fnt, fill = 'black')
img_draw.text((x+120, y), self.rundle + self.season_rank, font = fnt, fill = 'black')
img_draw.text((x+190, y), self.season_wins, font=fnt, fill='black')
img_draw.text((x+250, y), self.season_losses, font=fnt, fill='black')
img_draw.text((x+300, y), self.season_ties, font=fnt, fill='black')
img_draw.text((x+350, y), self.season_MPD, font=fnt, fill='black')
img_draw.text((x+420, y), self.season_TCA, font=fnt, fill='black')
img_draw.text((x+490, y), self.season_def, font=fnt, fill='black')
img_draw.text((x+560, y), self.last, font=fnt, fill='black')
# List of hold the players
# Appends each player to the list
Players = []
Players.append(Player('TestA'))
Players.append(Player('TestB'))
Players.append(Player('TestC'))
Players.append(Player('TestD'))
Players.append(Player('TestE'))
Players.append(Player('TestF'))
Players.append(Player('TestG'))
Players.append(Player('TestH'))
Log in to LL and scrape the data
This is where it starts to get complicated. Start by creating a Selenium driver and telling it where you installed the chrome (or other browser if you prefer) driver that allows it to be used to automate. Then we tell it the website to go to, in this case learnedleague.com. And then we tell it to find the element that is named username and fill in my username, which has been edited out below for obvious reasons. Then tell it to enter the password and then find and click the login button. You could also just have it hit enter after the password but I wanted to try a button, so I did.
# Create the webpage driver, navigate to learnedleague, and log in as matt
driver = webdriver.Chrome(executable_path = '/usr/local/bin/chromedriver')
driver.implicitly_wait(30)
driver.get("https://www.learnedleague.com")
typer = driver.find_element_by_name('username')
typer.send_keys('Enter_your_username_here')
typer = driver.find_element_by_name('password')
typer.send_keys('Enter_your_password_here')
button = driver.find_element_by_name('login')
button.click()
After it has logged in, the program then iterates through the players in the list and searches for each of them on the LL site. Once on their page, it finds the link to their current rundle and strips out the 8th letter of the link to store as the rundle name (A-E or R). Then, it finds all of the elements with a specific class name (std-mid… so intuitive, right?) and puts those in a list. I know that item 0 is the wins, 1 is the losses, etc. so it pulls each of them out and stores it in the player’s variables. Since they are already text, I just keep them that way. The only one that needs to be something else is the rank for sorting so I’ll just convert it as needed later. The final thing it does it find out the result of their last match, which is also in the list of std-mid items. We know that it will be the last letter that is in that list. So we iterate through it from the back going forward one each time and if it’s a letter, we stop and store that in the last variable.
# Iterate through each of the players and for each one search for their name
# then get the stats and store in list then pull out specific elements and store
# in the players stats
for i in Players:
# Find the player search bar and type in the player name and enter
typer = driver.find_element_by_id('f-player-search')
typer.send_keys(i.name + Keys.RETURN)
# Find the link showing the player's current rundle and extract the letter
typers = driver.find_element_by_class_name('std-left-key')
i.rundle = typers.text[7] + '-' #add a dash for presentation later
# Get all elements with class name 'std-mid', which includes the stats
# we are looking for among others, and store in a list
typers = driver.find_elements_by_class_name('std-mid')
# Find the specific elements and store in the player's variables
i.season_wins = typers[0].text
i.season_losses = typers[1].text
i.season_ties = typers[2].text
i.season_points = typers[3].text
i.season_MPD = typers[4].text
i.season_TCA = typers[6].text
i.season_def = typers[11].text
i.season_rank = typers[15].text
# The results of the last match are stored in the list above
# The last match will be the last letter in the list
# So we iterate backwards until we find a letter
for k in range(len(typers)-1, 1, -1):
if typers[k].text.isalpha():
i.last = typers[k].text
break
# Close the web driver -- very important
driver.close()
Sort the Players
Next up is to sort the players based on rundle and rank. Rundle A is higher than Rundle B, etc, etc down to Rundle E, with Rundle R at the bottom (because it’s the rookie rundle for a players first season). There are many ways to sort the players – I did it in one of the least efficient ways possible, but there are 8 people so it really doesn’t matter and makes it more readable. I start by making a new List for each rundle and then putting the players in there. Right now I’m only doing rundles C, D, E, and R since I can’t imagine knowing somebody smart enough to be in B or A.
# Divide the players into rundles rundleC = [] rundleD = [] rundleE = [] rundleR = [] for i in Players: if 'C' in i.rundle: rundleC.append(i) if 'D' in i.rundle: rundleD.append(i) if 'E' in i.rundle: rundleE.append(i) if 'R' in i.rundle: rundleR.append(i) # Sort each rundle using sortRundle method which takes a rundle as an argument # and modifies it to be sorted based on rank sortRundle(rundleR) sortRundle(rundleE) sortRundle(rundleD) sortRundle(rundleC)
Then I pass each of those individual rundles to the rundle sort function below. Against, this isn’t super efficient but it works and is easy to read. For the changes you make in the function to make it back to the rundles you passed it, it’s important to sue the i and j as an index with range rather than an object.
# Function to sort the rundle # Takes a rundle (list of players) as an argument and modifies that list # Uses a slow bubble sort but doesn't really matter def sortRundle(rundle): if len(rundle) > 1: for i in range(len(rundle)): for j in range(i,len(rundle)): if int(rundle[i].season_rank) > int(rundle[j].season_rank): rundle[i], rundle[j] = rundle[j], rundle[i]
Create an image from sorted data
Next up is to create the image from this data. Start by creating a blank image and choosing a font. Then, I make a new Player called the header. This allows me to use the Player class draw stats function to make the header rather than doing it by hand. I draw the header then draw a line under it. Through the drawing, I use a y_pos variable to keep track of the height each line should be drawn at. This make it easy to change the spacing to get a specific look and change it later on.
# Create blank image
blank_image = Image.new('RGBA', (625, 300), 'white')
img_draw = ImageDraw.Draw(blank_image)
fnt = ImageFont.truetype('/Library/Fonts/Arial.ttf', 18)
# Create a new 'Player' to store the header data
# Making it use the same Player class just saves time/space
Header = Player('Name')
Header.season_rank="Rank"
Header.season_wins="Ws"
Header.season_losses="Ls"
Header.season_ties="Ts"
Header.season_MPD="MPD"
Header.season_TCA="TCA"
Header.season_def=" DE"
Header.last="Last"
# y_pos will keep a record of the vertical position to type the stats
y_pos = 25
# Draw the header text on the image
Header.printStats(10,y_pos, img_draw)
img_draw.line((10, y_pos+25, 615, y_pos+25), fill=0)
y_pos+=30
Then, I iterate through each rundle and draw the items in order.
# Draw the players stats on the image # Starting with the higher rundles, incrementing the y_pos each time for i in rundleC: i.printStats(10,y_pos, img_draw) y_pos+=25 for i in rundleD: i.printStats(10,y_pos, img_draw) y_pos+=25 for i in rundleE: i.printStats(10,y_pos, img_draw) y_pos+=25 for i in rundleR: i.printStats(10,y_pos, img_draw) y_pos+=25
Now comes the interesting part. It’s easy to save the image to a hard drive location. But when uploading to the Groupme website, it’s very difficult to upload an image from the harddrive because it opens a system menu instead of a browser menu. So instead, I need to get the image onto my clipboard so that it can be pasted into the groupme chat. Again, this is actually pretty difficult. My work-around was to save the image to the desktop, then open the image in the default viewer (Preview on mac), copy it there using the pyautogui library to emulate a Command-C key press, then quit the app using the Command-Q key press. Now it’s in the clipboard and can be uploaded. If you haven’t played with pyautogui yet, I highly recommend it.
# Save image to the desktop
blank_image.save('/Users/mwwalk/Desktop/pic.png')
#blank_image.show()
# Open the saved image in default program (uses subprocess library)
subprocess.call(['open', '/Users/mwwalk/Desktop/pic.png'])
# Copy the image using pyautogui to emulate keyboard keys, then quit image program
pyautogui.hotkey('command', 'c')
pyautogui.hotkey('command', 'q')
Upload the image to the Groupme chat
Now, we need to copy the image in the clipboard to the groupme chat window. Start again by creating a driver and giving it the chrome path. Then go to the groupme signin page. Look for the phone number box and type in the number, then look for the password box and type in the password and hit enter.
# Create new webdriver and open groupme signin page
driver = webdriver.Chrome(executable_path = '/usr/local/bin/chromedriver')
driver.implicitly_wait(30)
driver.get("https://web.groupme.com/signin")
# Find the username textbox and type in phonenumber
# Then find password textbox and type in password
# Sleeps are necessary because groupme sucks
typer = driver.find_element_by_id('signinUserNameInput')
typer.send_keys('enter_phone_here')
time.sleep(1)
typer = driver.find_element_by_id('signinPasswordInput')
typer.send_keys('enter_password_here')
time.sleep(1)
typer.send_keys(Keys.RETURN);
Next, we need to find the right chat to click on. It’s not necessarily the only chat or even the top chat. There’s no class or id that is unique so we instead search using xpath with the title of the chat. Now we just need to click on the text box, which we can find by class name and hit command-v to paste. Oh, crap! That doesn’t work. Why doesn’t that work? Spend two hours searching and still can’t find out why it doesn’t work but find a workaround of using “SHIFT” + “INSERT” to paste instead. Whew! Pasting this takes us to an image upload page where we can put in captions. We just need to click the ‘send’ button, which we find using xpath. Now, we exit the browser and we’re done!
# Find the correct group chat and click it
# Sleep necessary because page loads slowly
driver.find_element_by_xpath('//*[@title="Oklahomies LL Chat"]').click()
time.sleep(5)
# Find the textbox and paste in the image copied earlier<span id="mce_SELREST_start" style="overflow:hidden;line-height:0;"></span>
# Normally you'd use Command-V to paste but for some reason that doesn't work
# so we use this weird workaround instead
typer = driver.find_element_by_class_name('emoji-wysiwyg-editor')
typer.send_keys(Keys.SHIFT, Keys.INSERT);
time.sleep(5)
# Find the send button on the image upload page and click it
button = driver.find_element_by_xpath("//button[text()='Send']")
#button.click()
time.sleep(5)
# Close the web driver -- very important
driver.close()
Complete Code
Here is the complete code. I tried to put enough comments in to explain everything without putting in so many as to be annoying. If you have any questions or comments please put them in the comments below and I’ll get back to you. Hope you find this helpful and as fun as I did!
# LearnedLeage Automated Poster
# Created by Matt Walker on July 7, 2018
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from PIL import Image, ImageDraw, ImageFont
import os
import subprocess
import pyautogui
import pyperclip
import time
# Player class to store stats and draw them in an image
class Player:
name = ""
rundle = ""
season_rank = ""
season_points = ""
season_wins = ""
season_losses = ""
season_ties = ""
season_MPD = ""
season_TCA = ""
season_def = ""
last = ""
def __init__(self, label):
self.name = label
# Function to make drawing stats easier, takes and x and y location and image
def printStats(self, x, y, img_draw):
img_draw.text((x, y), self.name, font = fnt, fill = 'black')
img_draw.text((x+120, y), self.rundle + self.season_rank, font = fnt, fill = 'black')
img_draw.text((x+190, y), self.season_wins, font=fnt, fill='black')
img_draw.text((x+250, y), self.season_losses, font=fnt, fill='black')
img_draw.text((x+300, y), self.season_ties, font=fnt, fill='black')
img_draw.text((x+350, y), self.season_MPD, font=fnt, fill='black')
img_draw.text((x+420, y), self.season_TCA, font=fnt, fill='black')
img_draw.text((x+490, y), self.season_def, font=fnt, fill='black')
img_draw.text((x+560, y), self.last, font=fnt, fill='black')
# Function to sort the rundle
# Takes a rundle (list of players) as an argument and modifies that list
# Uses a slow bubble sort but doesn't really matter
# It is important to alter the rundle as rundle[i]= so that the changes
# are maintained after the function exits
def sortRundle(rundle):
if len(rundle) > 1:
for i in range(len(rundle)):
for j in range(i,len(rundle)):
if int(rundle[i].season_rank) > int(rundle[j].season_rank):
rundle[i], rundle[j] = rundle[j], rundle[i]
# List of hold the players
# Appends each player to the list
Players = []
Players.append(Player('TestA'))
Players.append(Player('TestB'))
Players.append(Player('TestC'))
Players.append(Player('TestD'))
Players.append(Player('TestE'))
Players.append(Player('TestF'))
Players.append(Player('TestG'))
Players.append(Player('TestH'))
# Create the webpage driver, navigate to learnedleague, and log in as matt
driver = webdriver.Chrome(executable_path = '/usr/local/bin/chromedriver')
driver.implicitly_wait(30)
driver.get("https://www.learnedleague.com")
typer = driver.find_element_by_name('username')
typer.send_keys('enterusername')
typer = driver.find_element_by_name('password')
typer.send_keys('enterpassword')
button = driver.find_element_by_name('login')
button.click()
# Iterate through each of the players and for each one search for their name
# then get the stats and store in list then pull out specific elements and store
# in the players stats
for i in Players:
# Find the player search bar and type in the player name and enter
typer = driver.find_element_by_id('f-player-search')
typer.send_keys(i.name + Keys.RETURN)
# Find the link showing the player's current rundle and extract the letter
typers = driver.find_element_by_class_name('std-left-key')
i.rundle = typers.text[7] + '-' #add a dash for presentation later
# Get all elements with class name 'std-mid', which includes the stats
# we are looking for among others, and store in a list
typers = driver.find_elements_by_class_name('std-mid')
# Find the specific elements and store in the player's variables
i.season_wins = typers[0].text
i.season_losses = typers[1].text
i.season_ties = typers[2].text
i.season_points = typers[3].text
i.season_MPD = typers[4].text
i.season_TCA = typers[6].text
i.season_def = typers[11].text
i.season_rank = typers[15].text
# The results of the last match are stored in the list above
# The last match will be the last letter in the list
# So we iterate backwards until we find a letter
for k in range(len(typers)-1, 1, -1):
if typers[k].text.isalpha():
i.last = typers[k].text
break
# Close the web driver -- very important
driver.close()
# Divide the players into rundles
rundleC = []
rundleD = []
rundleE = []
rundleR = []
for i in Players:
if 'C' in i.rundle:
rundleC.append(i)
if 'D' in i.rundle:
rundleD.append(i)
if 'E' in i.rundle:
rundleE.append(i)
if 'R' in i.rundle:
rundleR.append(i)
# Sort each rundle using sortRundle method which takes a rundle as an argument
# and modifies it to be sorted based on rank
sortRundle(rundleR)
sortRundle(rundleE)
sortRundle(rundleD)
sortRundle(rundleC)
# Create blank image
blank_image = Image.new('RGBA', (625, 300), 'white')
img_draw = ImageDraw.Draw(blank_image)
fnt = ImageFont.truetype('/Library/Fonts/Arial.ttf', 18)
# Create a new 'Player' to store the header data
# Making it use the same Player class just saves time/space
Header = Player('Name')
Header.season_rank="Rank"
Header.season_wins="Ws"
Header.season_losses="Ls"
Header.season_ties="Ts"
Header.season_MPD="MPD"
Header.season_TCA="TCA"
Header.season_def=" DE"
Header.last="Last"
# y_pos will keep a record of the vertical position to type the stats
y_pos = 25
# Draw the header text on the image
Header.printStats(10,y_pos, img_draw)
img_draw.line((10, y_pos+25, 615, y_pos+25), fill=0)
y_pos+=30
# Draw the players stats on the image
# Starting with the higher rundles, incrementing the y_pos each time
for i in rundleC:
i.printStats(10,y_pos, img_draw)
y_pos+=25
for i in rundleD:
i.printStats(10,y_pos, img_draw)
y_pos+=25
for i in rundleE:
i.printStats(10,y_pos, img_draw)
y_pos+=25
for i in rundleR:
i.printStats(10,y_pos, img_draw)
y_pos+=25
# Save image to the desktop
blank_image.save('/Users/mwwalk/Desktop/pic.png')
#blank_image.show()
# Open the saved image in default program (uses subprocess library)
subprocess.call(['open', '/Users/mwwalk/Desktop/pic.png'])
# Copy the image using pyautogui to emulate keyboard keys, then quit image program
pyautogui.hotkey('command', 'c')
pyautogui.hotkey('command', 'q')
# Create new webdriver and open groupme signin page
driver = webdriver.Chrome(executable_path = '/usr/local/bin/chromedriver')
driver.implicitly_wait(30)
driver.get("https://web.groupme.com/signin")
# Find the username textbox and type in phonenumber
# Then find password textbox and type in password
# Sleeps are necessary because groupme sucks
typer = driver.find_element_by_id('signinUserNameInput')
typer.send_keys('enterphonenumberhere')
time.sleep(1)
typer = driver.find_element_by_id('signinPasswordInput')
typer.send_keys('enterpasswordhere')
time.sleep(1)
typer.send_keys(Keys.RETURN);
# Find the correct group chat and click it
# Sleep necessary because page loads slowly
driver.find_element_by_xpath('//*[@title="Oklahomies LL Chat"]').click()
time.sleep(5)
# Find the textbox and paste in the image copied earlier
# Normally you'd use Command-V to paste but for some reason that doesn't work
# so we use this weird workaround instead
typer = driver.find_element_by_class_name('emoji-wysiwyg-editor')
typer.send_keys(Keys.SHIFT, Keys.INSERT);
time.sleep(5)
# Find the send button on the image upload page and click it
button = driver.find_element_by_xpath("//button[text()='Send']")
#button.click()
time.sleep(5)
# Close the web driver -- very important
driver.close()
The Wandering Engineer 
Any chance you have an extra invite? I played Reach for the Top all through high school, Quiz Bowl in my undergrad, and attend weekly trivia. Just can’t seem to find someone I know who has one.