Distributed Storage Platform for Tobacco Control System Hive Tobacco Control · Thesis Simulation

3-Node Simulated Online Not a Real Cluster

THESIS RUNTIME / 2024 TRACE 0001 · READY

DISTRIBUTED STORAGE / Thesis System Reproduction

Turn a single scan,
into a verifiable data trail. From cigarette pack QR code to Hive user table, reproduce collection, decoding, deduplication, dual-replica storage, and querying entirely in the browser.

Tobacco package QR code sample photographed for the thesis project — Thesis Real-Photo Input`image/jpeg · QR target`

01SCANCigarette Pack Image
02DECODEOpenCV
03VERIFYSHA-256
04REPLICATEHDFS × 2
05QUERYHive / PyHive

2.9.2 Hadoop 3.1.2 Hive 3 nodes 1 Master, 2 Slaves

Front-end interactive demo; does not connect to real Hadoop/Hive services

Design of a Distributed Storage Platform for a Hive-Based Tobacco Control System

Enter the simulated workbench to manually complete a tobacco product scan registration and trace how data flows into HDFS and Hive.

Start Simulated Session

LOCAL BROWSER STATE

READY

Browser simulation environment · Not connected to a real Hadoop / Hive cluster

SESSION / LOCAL STORAGE SHA-256 / WEB CRYPTO NO SERVER WRITE

INGESTApache / PHP DECODEOpenCV VERIFYSHA-256 STOREHDFS × 2 QUERYHive 3.1.2

All User Data 0 Hive Simulation Records

Tobacco Types 0 Unique Count by ID

Today's Scans 0 Aggregate by date field

HDFS Active Nodes 3 / 3 Simulated Page Status

Welcome to the Tobacco Control System

admin

Upload a cigarette pack QR code image to register; view your personal data under "My Scans." Admins can view global statistics and manage users.

Personal Information

Paper Homepage Features

Username: admin
Account Type: Administrator

Recent Registrations

Upload Cigarette Pack QR Code Image

JPG · PNG · GIF / 5 MB

Paper Data Processing Pipeline

Waiting for Image

1PHP Image ReceptionFormat Validation, Timestamp Naming
2OpenCV DecodingPython QR Content Recognition
3Hash DeduplicationCheck Code and Hash Existence
4CSV Upload to HDFSWrite User-Specific Data Files
5LOAD DATA into HiveFields: id / code / hash / date / user

Recent Recognition Results

My Scan Data

0 records Tobacco Types 0 Hive mock query 0.00s

ID / Tobacco name	Code / QR content	Hash	Date	User	Status

Namenode informationHadoop 2.9.2 · hadoop1

Simulated interface, not connected to a real cluster

ACTIVEhttp://192.168.18.128:50070/dfshealth.html#tab-overviewNameNode · hadoop1

Started:	Sun May 26 11:07:48 +0800 2024
Version:	2.9.2, r826afbeae31ca687bc2f8471dc841b66ed2c6704
Compiled:	2018-11-13T12:14Z by root from release-2.9.2
Cluster ID:	CID-cc311286-df01-4af0-81b1-a5e8a7e17652
Block Pool ID:	BP-2028085156-192.168.18.128-1713836792385

Security is off.

Safemode is off.

610 files and directories, 192 blocks = 802 total filesystem object(s).

Configured Capacity:	53.09 GiB
DFS Used:	25.38 MiB (0.05%)
Non DFS Used:	6.12 GiB
DFS Remaining:	46.94 GiB (88.42%)
Block Pool Used:	25.38 MiB (0.05%)
DataNodes usages% (Min/Median/Max/stdDev):	0.04% / 0.05% / 0.07% / 0.01%
	3 (Decommissioned: 0)
Dead Nodes	0 (Decommissioned: 0)
Total Datanode Volume Failures	0 (0 B)
Number of Under-Replicated Blocks	0

In service
Down
Decommissioned

Node	Http Address	Last contact	Last Block Report	Capacity	Blocks	Block pool used	Version
hadoop1	192.168.18.128:50075	1 sec	37 sec	17.70 GiB	64	7.20 MiB (0.04%)	2.9.2
hadoop2	192.168.18.129:50075	2 sec	42 sec	17.70 GiB	64	8.60 MiB (0.05%)	2.9.2
hadoop3	192.168.18.130:50075	2 sec	45 sec	17.69 GiB	64	9.58 MiB (0.07%)	2.9.2

Node	Failed Storage Location	Failure Date	Estimated Capacity Lost
No entries to show.

Path	Snapshot Number	Snapshot Quota	Modification Time	Permission	Owner	Group
`/user/hive/warehouse`	3	65,536	2024-05-26 10:42	drwxrwxr-x	hive	supergroup

hdfs://hadoop1:9000/

Permission	Owner	Group	Size	Replication	Block Size	Name

ResourceManagerHadoop 2.9.2 · hadoop1

Simulated interface, not connected to a real cluster

STARTEDhttp://192.168.18.128:8088/cluster/clusterResourceManager · hadoop1

Hadoop

ResourceManager

Logged in as: dr.who

Cluster Metrics

Apps Submitted	Apps Pending	Apps Running	Apps Completed	Containers Running	Memory Used	Memory Total	VCores Used	VCores Total	Active Nodes	Unhealthy Nodes
13	0	1	12	2	2 GiB	24 GiB	2	24	3	0

Cluster Overview

Cluster ID:	1716710801674
ResourceManager state:	STARTED
ResourceManager HA state:	ACTIVE
ResourceManager started on:	Sun May 26 11:06:41 +0800 2024
ResourceManager version:	2.9.2 from 826afbeae31ca687bc2f8471dc841b66ed2c6704
Hadoop common version:	2.9.2
Scheduler type:	Capacity Scheduler

Nodes of the cluster

Node HTTP Address	Rack	State	Node Labels	Containers	Mem Used	Mem Avail	VCores Used	VCores Avail	Version
hadoop1:8042 192.168.18.128	/default-rack	RUNNING	<DEFAULT_PARTITION>	1	1 GiB	7 GiB	1	7	2.9.2
hadoop2:8042 192.168.18.129	/default-rack	RUNNING	<DEFAULT_PARTITION>	1	1 GiB	7 GiB	1	7	2.9.2
hadoop3:8042 192.168.18.130	/default-rack	RUNNING	<DEFAULT_PARTITION>	0	0 B	8 GiB	0	8	2.9.2

Applications

ID	User	Name	Application Type	Queue	StartTime	FinishTime	State	FinalStatus	Progress	Tracking UI
`application_1716710801674_0013`	hive	Hive SQL: tobacco summary	MAPREDUCE	default	10:12:16	N/A	RUNNING	UNDEFINED	68%	ApplicationMaster
`application_1716710801674_0012`	admin	Daily hash verification	MAPREDUCE	default	09:42:02	09:42:51	FINISHED	SUCCEEDED	100%	History
No applications match the selected state.

HiveServer2Apache Hive 3.1.2 · hadoop1

Simulated interface, not connected to a real cluster

STARTEDhttp://192.168.18.128:10002/Web UI 10002 · PyHive Thrift 10000

Active Sessions

User Name	IP Address	Operation Count	Active Time (s)	Idle Time (s)
admin	192.168.18.128	1	9,482	8
sutao	192.168.18.131	0	7,024	134
analytics	192.168.18.128	1	5,105	0
Total number of sessions: 3

Open Queries

User Name	Query	Execution Engine	State	Opened Timestamp	Opened (s)	Latency (s)	Drilldown Link
analytics	`SELECT user, COUNT(*) FROM scans GROUP BY user`	mr	RUNNING	Sun May 26 10:02:41 CST 2024	12	Not finished	Drilldown
Total number of queries: 1

Last Max 25 Closed Queries

User Name	Query	Execution Engine	State	Opened (s)	Closed Timestamp	Latency (s)	Drilldown Link
admin	`SELECT id, COUNT(*) FROM sutao GROUP BY id`	mr	FINISHED	2	Sun May 26 09:42:51 CST 2024	1	Drilldown
liuchao	`SELECT * FROM liuchao ORDER BY date DESC LIMIT 20`	mr	FINISHED	1	Sun May 26 09:35:18 CST 2024	0	Drilldown
Total number of queries: 2

Software Attributes

Attribute Name	Value	Description
Hive Version	3.1.2	Hive version and revision
Hive Compiled	2020-08-24, root	When Hive was compiled and by whom
HiveServer2 Start Time	Fri May 24 19:48:12 CST 2024	Date stamp of when this HiveServer2 was started
Thrift Service	hadoop1:10000	PyHive / Beeline connection endpoint

User Name	IP Address	Operation Count	Active Time (s)	Idle Time (s)	Default Database
admin	192.168.18.128	1	9,482	8	tobacco_control
sutao	192.168.18.131	0	7,024	134	tobacco_control
analytics	192.168.18.128	1	5,105	0	default
Total number of sessions: 3

Open Queries

User Name	Query	Execution Engine	State	Opened Timestamp	Latency (s)	Drilldown Link
analytics	`SELECT user, COUNT(*) FROM scans GROUP BY user`	mr	RUNNING	Sun May 26 10:02:41 CST 2024	Not finished	Drilldown

Last Max 25 Closed Queries

User Name	Query	Execution Engine	State	Closed Timestamp	Latency (s)
admin	`SELECT id, COUNT(*) FROM sutao GROUP BY id`	mr	FINISHED	Sun May 26 09:42:51 CST 2024	1
liuchao	`SELECT * FROM liuchao ORDER BY date DESC LIMIT 20`	mr	FINISHED	Sun May 26 09:35:18 CST 2024	0

Read-only simulation. This query form is added for the project demonstration and is not part of the native HiveServer2 3.1.2 Web UI.

Query engine ready · read-only simulation

Design of a Distributed Storage Platform for a Hive-Based Tobacco Control System

Distributed Data PlatformHadoop / HiveBrowser interaction reproduction

Using the cigarette pack QR code as the data entry point, each on-site scan is converted into a distributed regulatory record that is deduplicable, queryable, and traceable.

The system serves tobacco retail users and regulatory administrators. The thesis implementation uses an HTML/JavaScript frontend, Apache/PHP APIs, and Python scripts; OpenCV decodes images and computes SHA-256 hashes, deduplicated CSV files are uploaded to HDFS, then loaded into Hive user tables via PyHive. MySQL stores Hive Metastore metadata.

Cluster topology: hadoop1 / 2 / 3 · one master, two slaves
Storage & Compute: Hadoop 2.9.2 · Hive 3.1.2
Replica Strategy: HDFS replication = 2
Core Fields: id · code · hash · date · user

Paper Deployment Environment3 VMware Virtual MachinesApache / PHP / Python / Hadoop / Hive / MySQL

Currently on Public DisplayBrowser-Based Interactive SimulationReproduces UI, data flow, and permissions without connecting to the paper's intranet services.

Six Core Features Implemented in the Paper

Role-based views for regular users and admins

01Log inIdentity verification via Hive user table

02Sign UpUsername uniqueness check and write to HDFS/Hive

03Scan RegistrationUpload Image and Scan QR Code

04My ScansQuery personal data by username

05Statistics managementCharts for categories, quantities, and dates

06User ManagementAdmin view and delete users

From QR scan to Hive user table

Corresponds to Section 6.5 of the paper on scanning

Real photos from the paper

Upload Cigarette Pack QR Code Image

Field capture / INDEX.HTML The browser previews JPG, PNG, or GIF; PHP renames the file with a timestamp, then calls check.py.

Illustration of an operator scanning and registering a package

InputCigarette pack image OutputHTTPS Code

Locate and decode QR code

PYZBAR + OPENCVdecode(image) Read first HTTPS code

SHA-256code + hash

Hash Deduplication

CHECK_DATA_EXISTSBlock write if same code or hash already exists

CSV Upload to HDFS

INSECURECLIENT.UPLOAD/data/sutao/sutao.csv, with dual copies distributed across nodes

Write to Hive user table

LOAD DATA INPATHPyHive connection userdata.sutao and load one row

Simulated data for this run / TRACE 2024-05-04Link ready

Recognition result: Golden Leaf (Hard Hongqiqu)
QR Code: HTTPS://Y2WM.CN/99.1000.1/AC1016977001HUG9MA95DK9A7FEVZEJJH
SHA-256: 76226b01008bd9dd291c1485190cd0eb0dc1fa9f40201ba98b96404bab05b076
Final location: hdfs:///data/sutao/sutao.csv → userdata.sutao

Key implementation from paper appendix

Excerpted from index.html and check.py, preserving original call relationships

index.htmlEnable submit after selecting image

<input type="file" name="image"
  accept="image/*" onchange="previewImage()">
<button type="submit" disabled>Submit</button>

check.pyOpenCV / pyzbar and SHA-256

image = cv2.imread(image_path)
decoded_objects = decode(image)
hash_value = hashlib.sha256(
    data.encode()
).hexdigest()

check.pyUpload to HDFS and load into Hive

client.upload(hdfs_path, local_csv_path,
    overwrite=True)
load_query = f"LOAD DATA INPATH
    '{hdfs_path}' INTO TABLE `{username}`"
cursor.execute(load_query)

One primary, two secondary, three nodes

Simulated in browser · No real hosts connected

MASTER · Primary NodeSIMULATED

hadoop1

192.168.18.128

NameNode
ResourceManager
HiveServer2
8 GiB / 8 vCores

WORKER · Secondary Node 01SIMULATED

hadoop2

192.168.18.129

DataNode
NodeManager
HDFS Replica
8 GiB / 8 vCores

WORKER · Secondary Node 02SIMULATED

hadoop3

192.168.18.130

DataNode
NodeManager
HDFS Replica
8 GiB / 8 vCores

Cluster Management Entry

All three interfaces are browser simulations

NameNode Management Console

HADOOP HDFS 192.168.18.128:50070/dfshealth.html

Capacity and Block Status
DataNode Nodes
HDFS File Browser

Open HDFS Console

ResourceManager Console

HADOOP YARN 192.168.18.128:8088/cluster/cluster

Resource Scheduling Overview
Application Execution Status
NodeManager Nodes

Open YARN Console

HiveServer2 Console

APACHE HIVE 192.168.18.128:10002

Sessions and Queries
Databases and User Tables
Interactive HiveQL

Open Hive Console

Paper System Data Interaction Model

MySQL as Hive Metastore

01Browser InterfaceHTML · JavaScript 02Web InterfaceApache · PHP 03Data ScriptsPython · OpenCV · PyHive 04Data WarehouseHive · MySQL Metastore 05Distributed StorageHDFS · hadoop1/2/3

idTobacco Name / Unique Identifier

codeQR Code HTTPS Content

hashIntegrity and Duplicate Validation

dateYYYY-MM-DD HH:MM:SS

userUsername Associated with Record

Performance Requirements from Paper

The following are design goals, not measured results from this front-end simulation.

≤ 3 seconds: Page Load Target
≤ 5 seconds: Data Query Target
≥ 1000: Concurrent Users Target
≤ 80%: CPU / Memory Utilization Target

Tobacco types and date statistics

Three-Node Deployment Topology from the Paper