MoonDream Object Pointer with HTML Display

This lab allows you to query MoonDream to detect objects within an image, and then to create an HTML page with the image and an X at the X/Y coordinates of each object detected.

point_result[“points”] returns a list of the coordinates of the objects detected. These numbers are from top and left and represent from 0-1. We multiply the results by 100 to get a percent value and can then use the value for the top and left CSS attributes.

We create a container DIV, and then add overlay DIV's with red X's to mark the objects found.

You need to use the 2B model for this project.

python3 -m pip install moondream

python3 -m pip install pillow

download 2B int 8 Model from: https://github.com/vikhyat/moondream

import moondream as md
from PIL import Image

model = md.vl(model='./moondream-2b-int8.mf.gz')

picture = './img3.jpeg'
image = Image.open(picture)
encoded_image = model.encode_image(image)

query = 'people'
point_result = model.point(encoded_image, query)
print("Points:", point_result["points"])

with open('moondream-point.html', 'w') as file:
    file.write('''
               <div style="position: relative; 
               height: 400px; 
               width: auto; 
               display: inline-block;">
               ''')
    file.write(f'''
               <img src="{picture}" 
               style="height: 100%; 
               display: block;">
               ''')
    for point in point_result["points"]:
        x = point['x'] * 100
        y = point['y'] * 100
        print(f"{x} -- {y}")
        file.write(f'''
                   <div style="color:red; 
                   position: absolute; 
                   top:{y}%; 
                   left:{x}%;"
                   >X</div>
                   ''')
    file.write('</div>')