Dedupe, or not dedupe – that is the question

Hi, There:

It has been a little while again. I have been pretty busy recently. Anyway, happy a nice Summer weekend!

When you index millions of millions of data, inevitably you can face duplicated data. Duplicate data doesn’t necessarily mean that two documents are identical. But it can simply mean they are essentially the same document for your business’ purpose.

You can certainly dedupe them before indexing into Solr. But it is not always easy since you would need to maintain the state of the criteria of each document somewhere. It is harder when you have tons of documents. For this, Solr provides a handy way to help you dedupe when indexing.

I am using a simple approach to illustrate how dedupe can be configured, and analyze how it works.

Considering a document below, which contains information of a person. The combination of the name, ssn4 and dob fields is assumed as the uniqueness indicator for a given person. The id field can be different, and the city is not part of the duplication criteria because people move around the country.

{
        "name":"John Doe",
        "ssn4":9999,
        "dob":"1995-07-04T00:00:00Z",
        "id":"1",
        "city":"san francisco"
}

If we want to index each person only once in Solr (based on name/ssn4/dob), we can set up in Solr like this.

In solrconfig.xml, add/enable the updateRequestProcessorChain.

     <updateRequestProcessorChain name="dedupe">
       <processor class="solr.processor.SignatureUpdateProcessorFactory">
         <bool name="enabled">true</bool>
         <str name="signatureField">signature1_s</str>
		 <bool name="overwriteDupes">true</bool>
         <str name="fields">name,dob,ssn4</str>
         <str name="signatureClass">solr.processor.Lookup3Signature</str>
       </processor>
       <processor class="solr.LogUpdateProcessorFactory" />
       <processor class="solr.RunUpdateProcessorFactory" />
     </updateRequestProcessorChain>

This request processor SignatureUpdateProcessorFactory will calculate a signature field with combination of name/dob/ssn4 field, and put in a new field called signature1_s. Make sure this new field signature1_s is defined in your schema. Lookup3Signature is the class that defines the algorithm to generate the signature hash. You could use others such as MD5.

Now you can index some data by curling. Note the update.chain=dedupe will enable the chain processor by it’s name, dedupe. Without this, the processors won’t run. You could make the dedupe process as defaults, then you would not need it in the parameter.

curl -X POST -H "Content-type:application/json" --data-binary @people.json "http://localhost:8983/solr/people/update/json?update.chain=dedupe&commit=true"

people.json

[
{"name":"John Doe","ssn4":9999,"dob":"1995-07-04T00:00:00Z", "id":"1","city":"san francisco"}
]

Check the indexed doc in Solr and see this. Note the generated signature field signature1_s with the value 286004b0d7fd7de4.

{
        "name":"John Doe",
        "ssn4":9999,
        "dob":"1995-07-04T00:00:00Z",
        "id":"1",
        "city":"san francisco",
        "signature1_s":"286004b0d7fd7de4",
        "_version_":1703663998959878144
}

Now let’s index a slightly different people.json buy changing the value of id (if we don’t change id, Solr will overwrite the document anyway), and city (assuming John Doe moved to new york). We shall keep the name/ssn4/dob fields intact.

New people.json

[
{"name":"John Doe","ssn4":9999,"dob":"1995-07-04T00:00:00Z", "id":"2","city":"new york"}
]

Check the indexed doc in Solr. We only see the new document, which overwrote the previous one since the signature field is the same. Note the signature1_s has the same value as the previous document, but other non-signature fields have changed.

{
        "name":"John Doe",
        "ssn4":9999,
        "dob":"1995-07-04T00:00:00Z",
        "id":"2",
        "city":"new york",
        "signature1_s":"286004b0d7fd7de4",
        "_version_":1703664476960587776
}

Now let’s turn off the dedupe flag and set overwriteDupes to false. Don’t forget to reload the core for this change.

<bool name="overwriteDupes">false</bool>

Try the same experiment all over again – you will see both documents, as expected. Even though the signature is the same, Solr indexed the two documents as expected.

{
        "name":"John Doe",
        "ssn4":9999,
        "dob":"1995-07-04T00:00:00Z",
        "id":"1",
        "city":"san francisco",
        "signature1_s":"286004b0d7fd7de4",
        "_version_":1703664823793876992
},
{
        "name":"John Doe",
        "ssn4":9999,
        "dob":"1995-07-04T00:00:00Z",
        "id":"2",
        "city":"new york",
        "signature1_s":"286004b0d7fd7de4",
        "_version_":1703664839278198784
}

An interesting thought to check is this. So far, these are dedupe or not-dedupe when new documents are indexed into Solr to overwrite existing documents. How about the documents are being indexed in the same commit? Consider if the input people.json like this, both documents are in the same commit:

[
{"name":"John Doe","ssn4":9999,"dob":"1995-07-04T00:00:00Z", "id":"1","city":"san francisco"},
{"name":"John Doe","ssn4":9999,"dob":"1995-07-04T00:00:00Z", "id":"2","city":"new york"}
]

And the result is .. exactly the same! Solr doesn’t care if the dedupe is within one or multiple transactions. It behaves just as the update chain processor is configured. This is consistent and nice.

This technique provided by Solr should conveniently help you dedupe documents based on your criteria of uniqueness. Personally, I was interested in seeing if there is any way to NOT index a document if there is already a duplicate document existing in the index. There are some use cases for that. For example, avoiding re-index a duplicate document could save resource in Solr, and avoid re-merging too. But it seems the default behavior of dedupe inside RunUpdateProcessorFactory is to overwrite instead of skipping. I think we can use some custom implementation to change this behavior, i.e. skip indexing if the calculated value of signature field already exists in Solr.

For more info, see https://solr.apache.org/guide/8_4/de-duplication.html

Cheers!

~T

Deep pagination’s slowness and OOM in Solr

Hi, There:

Finally it gets a bit like Spring in the northeast. Also, a friendly reminder – tax return due day is 5/17/21.

So Solr has a built in pagination feature with parameters like start (starting document) and rows (page size). It is intuitively easy to understand and nice to use. But it doesn’t come without some caveats. Imagine what Solr is doing behind the theme. Solr first sorts the matched documents in memory. Then it moves to the starting document using the parameter start; then fetch the next rows number of document before returning to the client. The key point here is Solr sorts and stores all the documents in its memory, at least up to the point where it includes the documents based on the start and rows.

Most of the use cases for Solr is to search small amount of hit documents based on custom queries, and return the top matched documents by score or sort. Rarely in real applications, people cares about anything after 10,000 documents. But in the case when client wants to do very “deep” pagination, ex, trying to get 1,000 documents starting from 1,000,000th, Solr needs to sort and save all of the 1,001,000 docs in memory. That’s when the query takes huge amount of time to run and often cause OutOfMemoryError. This could bring Solr service down if not protected by some layer.

I did some tests on my local Solr core with 11+ million docs and 512 MB of JVM memory. I altered the two parameters, start and rows. The result is shown here:

Both start and rows have tremendous impact on the query time, independently. And they seem to have similar rate of impact (I am not to prove it here). More importantly, though not shown in the graph, my Solr got OOM when either start and rows reaches above 2,000,000. Therefore, deep pagination is both dangerous (OOM) and impractical (slowness).

So how to deal with this problem if client wants to get deep paginated documents. In my testing case, for example, to get anything after 5,000,000th document? The solution Solr provides is CursorMark. Let’s see some code first.

int rows = 1000000;
SolrQuery query = new SolrQuery();
query.setQuery("*:*");
query.setRows(rows);
query.setSort("id", ORDER.asc);
		
String cursorMark = CursorMarkParams.CURSOR_MARK_START;
int count = 0;
boolean noMore = false;
while (!noMore) {

	long start = System.currentTimeMillis();		
	query.set(CursorMarkParams.CURSOR_MARK_PARAM, cursorMark);
	QueryResponse response = solr.query(query);
	String nextCursorMark = response.getNextCursorMark();
	int pageSize = response.getResults().size();
	count += pageSize;
	if (cursorMark.equals(nextCursorMark)) noMore = true;
    cursorMark = nextCursorMark;
	System.out.println("Total " + count + ". This page has " + pageSize + " costs " + (System.currentTimeMillis() - start) + " ms. Next Cusor " + cursorMark);
}

Solr provides you with a CursorMark to help you progressively go deeper and deeper based on your query. If you have a consistent sort parameter, every time when rows of documents are returned, there is a nextCursorMark in the response. This nextCursorMark needs to be returned back to the Solr in the next query. Solr will use it to “calculate” which document to skip and start with the one after it for the next batch of rows. In essence, one can think of this nextCursorMark as a filter-query-like parameter, Solr will “filter” out all documents that has the calculated property “less than and equal to” this CursorMark. It goes this way one page by page, and when no more documents are in the page, Solr will return the last nextCursorMark again, to let the client know there are no more documents.

Output of the SolrJ code:

Total 1000000. This page has 1000000 costs 71220 ms. Next Cusor AoE/BTE3MmJiOWViLTg3Y2MtNGJjMy04YzMzLWNlYWY0MzIxZDk0Mw==
Total 2000000. This page has 1000000 costs 68629 ms. Next Cusor AoE/BTJlNGNhYmU4LTVjMjYtNGYwMy05MTUxLTA2YTNlNzk5ZGQ0Yw==
Total 3000000. This page has 1000000 costs 64734 ms. Next Cusor AoE/BTQ1NzBjYjIwLWJjNWUtNGU1MC04Mjc5LTg5YzM5Zjk2MTdmMg==
Total 4000000. This page has 1000000 costs 63366 ms. Next Cusor AoE/BTVjOTllZDExLWJmNTEtNGVjMS1hMDNkLTYwN2RkMTIzMTQ1YQ==
Total 5000000. This page has 1000000 costs 62166 ms. Next Cusor AoE/BTczYzY0MmRkLTgxYWItNGJjNy05YmFiLWYwNWZkNDgyODBhNw==
Total 6000000. This page has 1000000 costs 61337 ms. Next Cusor AoE/BThhZjEwY2UyLTI0YWYtNGNmZS04ZmUxLWFiMWFjMWQ2Zjk5Zg==
Total 7000000. This page has 1000000 costs 57545 ms. Next Cusor AoE/BWEyMTE0NTE1LTNjMjEtNGU0ZS05MjA2LTAyYTM5MTY3MjM1MA==
Total 8000000. This page has 1000000 costs 54268 ms. Next Cusor AoE/BWI5M2M1ODZkLWQwNGYtNDA2ZC1iYmRhLWQ5ZGE5NjRiY2E2Mw==
Total 9000000. This page has 1000000 costs 53344 ms. Next Cusor AoE/BWQwNzIxZTM5LTJkZGEtNGE4NS1hOWY5LTNmM2VlNmJlZGFhMA==
Total 10000000. This page has 1000000 costs 49601 ms. Next Cusor AoE/BWU3YTY1NDcxLTk5MWQtNDgyNi05Y2NlLWEyYWI1YTQ0NjhmZg==
Total 11000000. This page has 1000000 costs 40592 ms. Next Cusor AoE/BWZlY2MzY2JmLTJlMGQtNGEwYy1iODVkLTQ5NzZmYmFiMjA1MQ==
Total 11051795. This page has 51795 costs 2202 ms. Next Cusor AoE/BWZmZmZmZjk4LWUyZDgtNGU3Zi1hYTllLTM4NmViZTBiNWY4Ng==
Total 11051795. This page has 0 costs 318 ms. Next Cusor AoE/BWZmZmZmZjk4LWUyZDgtNGU3Zi1hYTllLTM4NmViZTBiNWY4Ng==

Each query for the 1,000,000 rows is very consistent in query time, around 4-7 seconds. This is because each query is basically an independent query. Also note the last two query has the same NextCursorMark, indicating the end of the documents.

This kind of cursor controlled pagination saves query time, since it sorts less number of documents (in almost like divider and conquer fashion). More critically, there will be no OOM, unless the rows parameter is outrageously too big.

One more requirement is the sorting field has to be unique in the collection. Why? this is because if the sorting field is not unique, there is some chance (depends on the occurrence rate of duplicated sort field) that the cursorMark is calculated from a document with one of the duplicated fields. Then the next query will not know which documents to skip. For example, if you want to paginate on documents based on first name, then if a common name like “tony” happens to be used to calculate the nextCursorMark, Solr can’t decide which “tony” to skip in the next query. Solr can certainly skip all “tony” documents, but then the query result will lose documents that should have been returned.

Deeper pagination is not that commonly needed in biz use cases, in my opinion. But if must, use Cursor Mark to speed up the query and more importantly, not to bring Solr to its knees by OOM.

Cheers!

~T

Kafka Streams process vertically

Hi, There:

Happy Friday! Can it be any colder one day before May 1st in the northeast? 😦

If you ever hear that Kafka Streams process vertically, it is actually more important than it sounds like. I am doing a simple demo below.

The Streams Pipeline:

final KStream<String, String> input = builder.stream(INPUT_TOPIC);

input
.peek((k, v) -> System.out.println(new Date() + " @peek 1 value: " +  v))
.filter((k, v) -> v.length() < 10)
.peek((k, v) -> System.out.println(new Date() + " @peek 2 value: " +  v))
.mapValues(v -> v = holdAndUpper(v))
.peek((k, v) -> System.out.println(new Date() + " @peek 3 value: " +  v))
.print(Printed.toSysOut());

Two meaningful processes, one to filter out anything longer than 10 chars, and mapValues to call an external function holdAndUpper. Between them, there are peek functions.

In the function holdAndUpper, we hold for n seconds and then return the uppercased string of the input. n is the length of the input String. If the input is 8 in length, it will wait for 8 seconds before returning.

private static String holdAndUpper(String s) {
	try { 
		Thread.sleep(s.length()*1000); 
	} catch (InterruptedException e) {
		e.printStackTrace(); 
	}
	return s.toUpperCase();
}

Now input all these three strings in one step.

abcdefghiklmn
abcdefg
ab

The Output:

Fri Apr 30 17:39:35 EDT 2021 @peek 1 value: abcdefghiklmn
Fri Apr 30 17:39:35 EDT 2021 @peek 1 value: abcdefg
Fri Apr 30 17:39:35 EDT 2021 @peek 2 value: abcdefg
Fri Apr 30 17:39:42 EDT 2021 @peek 3 value: ABCDEFG
[KSTREAM-PEEK-0000000005]: null, ABCDEFG
Fri Apr 30 17:39:42 EDT 2021 @peek 1 value: ab
Fri Apr 30 17:39:42 EDT 2021 @peek 2 value: ab
Fri Apr 30 17:39:44 EDT 2021 @peek 3 value: AB
[KSTREAM-PEEK-0000000005]: null, AB

Under this single processor condition, a few important things can be observed from the output.

• The filter of > 10 chars stops right away and it finished life inside of the processor at the filter. It is immediately followed by the second input to the steam, which has 7 chars.
• While the 7 chars string (abcdefg) in the processing pipeline, the third input (ab) never enters the pipeline. The 7 chars string took all the expected time of 7 seconds before it exits the processor.
• The third 2 chars string (ab) only enters the processor after the 7 chars string is finished.

Now, that is the meaning of “vertical processing” of Kafka Streams. It also emphasizes the importance of order of the processing steps. For one, the filter step should be put in the front of the pipeline. If a time-costly step is before a filter, an input could be going through that step and later be filtered off in the filter step. That is not only a waste of the process itself, other inputs could be waiting at the same time for bigger loss.

Depends on the nature of the data processing, the processes can be spread across horizontally to many consumers, or vertically to multiple threads. But just remember in each instance of the processor, the steps are going through vertically for each input.

Cheers!

~T

Recount the Pi digits by Java Streams

Hi, There:

A while back, I used CyclicBarrier to count Billons of digit of Pi. https://tonyyan.wordpress.com/2017/09/28/mimic-mapreduce-using-cyclicbarrier-to-count-a-billion-digits-of-pi/ Today, let’s try to use a much simpler way to do similar things, by Java Stream. Here is the code. The idea is to map the bytes read into a count Map of each digits from 0-9, and then reduce to a single Map of the same. Indeed, the stream API made the code very concise and easy to understand.

Another key benefit is that we can use parallel Stream and custom forked threads to tune the optimal thread numbers. The multi-threading is totally under the hood.

public class CountPi {

    public static void main(String[] args) throws IOException, ExecutionException, InterruptedException {

        long start = System.currentTimeMillis();
        List<byte[]> PiList = readPi(new File("src/main/resources/pi-billion.txt"));

        System.out.println("read time " + (System.currentTimeMillis() - start));
        start = System.currentTimeMillis();

        ForkJoinPool customThreadPool = new ForkJoinPool(32);

        Optional<Map<Byte, Integer>> c = customThreadPool.submit(
                () ->PiList.parallelStream().map(b -> countDigit(b)).reduce((m, n) -> mergeMap(m,n)))
                .get();

        System.out.println(c);
        System.out.println("stream time " + (System.currentTimeMillis() - start));

    }

    private static List<byte[]> readPi(File f) throws IOException {

        InputStream input = new FileInputStream(f);
        List<byte[]> PiList = new ArrayList<>();
        byte[] buffer = new byte[1024*1024];
        int data = input.read(buffer);
        while(data != -1) {
            PiList.add(buffer);
            data = input.read(buffer);
        }
        input.close();
        return PiList;

    }
    private static Map<Byte, Integer> countDigit(byte[] bytes){
        Map<Byte, Integer> map = new HashMap<>();
        for (byte b:bytes) map.put(b, map.getOrDefault(b,0)+1);
        return map;
    }

    private static Map<Byte, Integer> mergeMap(Map<Byte, Integer> m, Map<Byte, Integer> n){
        for (Byte b:n.keySet()){
            m.put(b, m.getOrDefault(b,0) + n.getOrDefault(b,0));
        }
        return m;
    }
}

Sample output:

read time 1315
Optional[{48=99304722, 49=100393236, 50=100106082, 51=99930546, 52=100190988, 53=99815112, 54=99920052, 55=100076508, 56=100064106, 57=100540152}]
stream time 6891

Looks like the ROI diminishes when the thread count is more than 4, due to overheads of threading cost.

Cheers!

~T

A Prototype of a Decentralized Email system on Ethereum

Hi, There:

How about an email system that is:

• Decentralized without servers (Ethereum)
• Very minimal spamming (Gas cost)
• 100% of privacy with anonymity (addresses and end to end encryption)
• Recipient can reject senders if desired (isn’t this nice!)

Solution: A decentralized email network on Ethereum with end to end email content encryption

The Smart Contract

// SPDX-License-Identifier: GPL-3.0
pragma solidity >=0.7.0 <0.8.0;
pragma experimental ABIEncoderV2;

contract Deemail {
    
    enum APPROVED_STATUS{UNKNOWN, APPROVED, REJECTED}
    struct Mail {address from; address to; uint time; string cypherMessage; }

    // mapping of mapping contains FROM=>To=>1 approved, 0, unknown -1 rejected
    mapping (address=>mapping(address=>APPROVED_STATUS)) approvedMap;
    
    // cypher email boxes contains TO => From => mail Struct[]
    mapping(address=>Mail[]) mailBoxes;
    
    // email received event
    event EmailReceivedEvent(address from, address to, uint time);
    
    constructor() { }
    
    function allowSend(address to) view public returns(bool) {
        if (msg.sender == to) return true;
        else if (approvedMap[msg.sender][to] == APPROVED_STATUS.UNKNOWN || approvedMap[msg.sender][to] == APPROVED_STATUS.APPROVED) return true;
        else return false;
    }
    
    /**
     *  Approve sending from FROM
     */
    function approve(address from) public {
        approvedMap[from][msg.sender] = APPROVED_STATUS.APPROVED;
    }
    
    /**
     *  Reject sending from FROM, and delete emails from FROM
     */
    function reject(address from) public {
        approvedMap[from][msg.sender] = APPROVED_STATUS.REJECTED;
        Mail[] storage mails = mailBoxes[msg.sender];
        for (uint i=0;i<mails.length;i++){
            if (from == mails[i].from) delete mails[i];
        }
    }
    
    /**
     *  send Email to a single TO address
     */    
    function sendEmail(address to, string memory cypherMessage) public {
        require(allowSend(to), "Emails to this recipient are not allowed!");
        mailBoxes[to].push(Mail(msg.sender, to, block.timestamp,cypherMessage));
        emit EmailReceivedEvent(msg.sender, to, block.timestamp);
    }
    
    // get all my Emails
    function retrieveEmails() public view returns(Mail[] memory) {
        return mailBoxes[msg.sender];
    }
}

The smart contract contains these functions. Any address can send email to any other addresses on the network. Emails are stored on the blockchain. Any recipient can reject the sender. If that happens, the sender can no longer send to the recipient, and emails sent by this sender are deleted from the chain. Recipient can only retrieve her own emails from the blockchain.

Future features may include one to many sends. Importantly on the client side, email content should be encrypted using recipient’s public key before the sender sends the email. The recipient will need to decrypt using private key to view the content of the email. Unencrypted emails would be visible to the whole network in theory, although only the recipient can retrieve her own emails in retrieveEmails() in the contract.

Some screenshot from the client side:

Send email from one address to another

Read your emails

After rejecting one sender

My client codes are not shown here. I used some react codes from https://github.com/dappuniversity/dbank/tree/starter_kit Thanks.

Cheers! ~T

Use JS to validate PoW on one BTC block

Hi, There!

Let’s try to validate a real bitcoin block’s POW using JavaScript. I omitted generation of Merkle Root from Transactions (maybe for another post). Used all data from this block: https://btc.com/0000000000000000000e5ac8accffaa7ba73e200354b799133a29464cac7b8a6

Very helpful reference: https://en.bitcoin.it/wiki/Block_hashing_algorithm

JS is as below, pretty self-explanatory.

const { assert } = require("console");
const CryptoJS = require("crypto-js");

// BTC #664,061 
// Block headers - needs change to bigendians
const CLAIMED_POW = '0000000000000000000e5ac8accffaa7ba73e200354b799133a29464cac7b8a6';
const version = '2000e000';
const prevBlock = '0000000000000000000c83a44631db885d262ff804ebde085f37f3d2d38521b9';
const merkleRoot = 'fee9c39d82f8c93a26f6ee1411ef16b83d9500249d50c73bbc91d313a6120bf3';
const timestamp = '2021-01-01 18:58:20'; // UTC needs to get Epoch in seconds
const bits = '170f2217';							
const nonce = 'cbf261da';

const epochSec = new Date(timestamp).getTime() / 1000;
const hexEpoch = epochSec.toString(16);

function endian(input){
   return Buffer.from(input, 'hex').reverse().toString('hex');
}

function getTarget(bits){
    let digits = 2*parseInt('0x' + bits.substring(0,2))-6;
    let target = bits.substring(2);
    for (let i=0;i<digits;i++) target += '0';
    return target;
}

function hashIt(input){
    var wordArray = CryptoJS.enc.Hex.parse(input);
    //console.log(wordArray); 
    var hash = CryptoJS.SHA256(wordArray);
    console.log('hash -->' + hash + '<--');
    return hash.toString();
}

const hexheaders = endian(version) + endian(prevBlock) + endian(merkleRoot) + endian(hexEpoch) + endian(bits) + endian(nonce);
console.log(hexheaders);

const hash = hashIt(hashIt(hexheaders));
const CALCULATED_POW = endian(hash);

console.log(CALCULATED_POW);

assert(CALCULATED_POW == CLAIMED_POW, 'Error, failed to validate the claimed POW');

// validate target
const target = getTarget(bits);
console.log('target ' + target);
console.log('pow    ' + CALCULATED_POW);
assert(parseInt(target, 16) > parseInt(CALCULATED_POW, 16), 'Error, target not reached');

Output:

Make any change to the nonce, or any other headers, should result in error, as expected. Like this:

Cheers! ~T

Where are my Contracts and Events on Ganache?

Hi, There!

Ganache is a great, almost one-click development Ethereum you can use on desktop. One thing many folks complain about is not being able to see Events and Contracts in Ganache when they are expected.

No events or contracts show up without truffle project defined

Actually, if a Truffle project is configured, the Contracts and Events will show up in the Ganache dashboard.

Add a truffle project and restart

Contracts are shown deployed after truffle migrate

Events are showing up after emitted

Details of an Event

These are very handy to get Contract and Events, a lot easier than using truffle console and web3.js.

contract MyContract {
  event MyEvent(address indexed from, string message);
  constructor() public {
  }
  function foo() public {
    emit MyEvent(msg.sender, 'hello foo');
  }
}

truffle-config.js should have interface and sol files directories, such as:

  contracts_directory: './src/contracts/',
  contracts_build_directory: './src/abis/',
  compilers: {
    solc: {
      version: ">=0.6.0 <0.8.0",
      optimizer: {
        enabled: true,
        runs: 200
      }
    }

Cheers! ~T

Calling contract from a contract on Ethereum

Hi, there:

One can call a contract from a contract on the Ethereum by the address. This allows function reusability. A simple example as below:

Contract A: Tickers contains company names and their stock tickers.

contract Tickers {
    
    mapping (string => string) tickers;
    using Utils for *;

    constructor (){
        tickers['AMAZON'] = 'AMZN';
        tickers['APPLE'] = 'AAPL';
        tickers['MERCK'] = 'MRK';
    }
    
    function getTicker(string memory n) public view returns (string memory t) { 
        return tickers[Utils._toUpperCase(n)];
    }
	
}

library Utils {
    
    function _toUpperCase(string memory str) public pure returns (string memory) {
		bytes memory bStr = bytes(str);
		bytes memory bOutput = new bytes(bStr.length);
		
		for (uint i = 0; i < bStr.length; i++) {
	                 bytes1 _b1 = bStr[i];
			 if (_b1 >= 0x61 && _b1 <= 0x7A) {
				bOutput[i] = bytes1(uint8(_b1) - 32);
			} else {
				bOutput[i] = _b1;
			}
		}
		return string(bOutput);
	}
}

Contract B: StockTest to use Contract A.

contract StockTest {
    
    Tickers private tickers = new Tickers();
    Tickers private contractTickers;

    function registerTickersContract(address a) public returns(bool) {
        contractTickers = Tickers(a);
        return true;
    }
 
    function getStock(string memory name) 
        public view returns(string memory, string memory) {
        
        string memory ticker;
        if (address(contractTickers) == address(0)){
            ticker = tickers.getTicker(name);  // in case address not set, call local
        } else {
            ticker = contractTickers.getTicker(name);
        }
        return (name, ticker);
    }
}

registerTickersContract() allows injection of Contract A’s address before Contract B can call it.

The client web3 code is straightforward:

        const abi = [....];
        
        const Web3 = require('web3');
        const url = 'http://localhost:8545/';
        const web3 = new Web3(new Web3.providers.HttpProvider(url));
        const callerAccount  = '0x0eCF7fED2d0ADEFb32CD87DF574289133c1e7D5f';
        const contractAddress = '0xF7992417f14D4020322621E2dDafC91DB4841037';

        // instantiate a new web3 Contract object
        let contract = new web3.eth.Contract(abi, contractAddress);

        contract.methods.registerTickersContract('0xD33c17761B537ecdfE7747725BC1F3D96BB63042')
        .call().then((result)=>{
            if (result) {
                contract.methods.getStock('amazon').call().then((result)=>{
                    console.log(result);
                });
            } 
        });

That’s pretty much it. Cheers!

~T

Subscribe and listen to events on Ethereum

Hi, There!

You would often want to monitor what’s going on with the contract on the Ethereum network, especially your own contract. For this, you need to do (1) emit some events on the network (2) subscribe to the event based on address and /or topics.

• Emit event in Solidity

pragma solidity >=0.7.0 <0.8.0;
contract LogEventExample {
    event MyEvent(address indexed a, uint256 v); // indexed will be in topic
    function triggerEvent() public payable {
        emit MyEvent(msg.sender, 1);
    }
}

• Make sure –ws is enabled if using geth. Websocket allows monitoring of the event after subscribe.

geth --datadir test-chain-dir --http --ws --dev --http.corsdomain "https://remix.ethereum.org,http://remix.ethereum.org"

• Subscribe by Web3

const Web3 = require('web3');
const url = 'ws://127.0.0.1:8546';
const web3 = new Web3(url);

var options = {
    address: '0xfbBE8f06FAda977Ea1E177da391C370EFbEE3D25',
    topics: [
        '0xdf50c7bb3b25f812aedef81bc334454040e7b27e27de95a79451d663013b7e17',
        //'0x0000000000000000000000000d8a3f5e71560982fb0eb5959ecf84412be6ae3e'
      ]
};

var subscription = web3.eth.subscribe('logs', options, function(error, result){
    if (!error) console.log('got result');
    else console.log(error);
}).on("data", function(log){
    console.log('got data', log);
}).on("changed", function(log){
    console.log('changed');
});

The subscribing options filter out events based on contract address, topics. If you want to subscribe to all events on the network, comment them out.

• call triggerEvent() on the contract in the network and watch the event show up in Web3

A detailed nice explanation about event data can be found in this article https://medium.com/mycrypto/understanding-event-logs-on-the-ethereum-blockchain-f4ae7ba50378

Cheers!

~T

Use web3.js to connect to Ethereum mainnet via Infura

Hi, There:

Long time again!

When you play with web3.js, if you want to connect to Ethereum mainnet quickly without using real account, you can use the Infura as the bridge to get into the ETH2. Infura provides a nice layer to manage/monitor the access to Ethereum.

A few main steps:

1. Go to Infura.io to register for free (with limited daily requests).
2. Create a new project called Dapp1, or whatever you like.
3. Get a https or websocket URL which comes with API token, such as:
   • https://mainnet.infura.io/v3/b0bb******
4. Run node.js code. Example as below:

const Web3 = require('web3');

const url = 'https://mainnet.infura.io/v3/b0bb******';

const web3 = new Web3(url);

// OR const web3 = new Web3(new Web3.providers.HttpProvider(url));

(async() => {   

try {        

const abi = [{ "constant": true, "inputs": ............ ];

const contractAddress = '0xdac17f958d2ee523a2206206994597c13d831ec7';

let contract = new web3.eth.Contract(abi, contractAddress);
  let name = await contract.methods.name().call();  

console.log(name);   

} catch (e) {       

console.log(e.message);

}  
})();

The output should be a value like Tether USD which is contract’s return value of function name().

Cheers!

~T